Job Details

Data Platform and Visualization Engineer
Job Number: 26-00174

Use your skills where innovative technology solutions begin. ECLARO is looking for a Data Platform and Visualization Engineer for our client in Los Altos, CA.

ECLARO's client is a leading technology solutions provider, collaborating with customers to manage their needs and achieve success in their business goals. If you're up to the challenge, then take a chance at this rewarding opportunity!

Position Overview:

• A contractor to help build and evolve our internal data platform that supports vehicle testing, experimentation, and machine learning workflows.
• Focuses on implementing and extending data ingestion pipelines, automated processing workflows, metrics tracking systems, and web-based visualization tools under the guidance of the team.
• Work with existing systems and well-defined components, contributing features and improvements that are used directly by researchers.
• The Contractor will provide engineering services to support the development and extension of internal data platform tooling supporting vehicle testing, experimentation, and machine learning workflows.
• The scope includes ownership and extension of existing systems, implementation of automated pipelines, development of web-based visualization tools, and delivery of data traceability mechanisms.

Pay Rate: $85.00-90.00/Hour

Responsibilities:

• Implement and extend data ingestion and processing workflows for large, heterogeneous datasets collected from vehicle tests and ML pipelines.
• Contribute to improving orchestration, scheduling, and reliability of long-running data workflows operating under real-world constraints.
• Integrate downstream automation such as metric computation, plotting, and LLM-based postprocess tooling.
• Implement backend services and APIs that support data indexing, metadata management, and experiment tracking.
• Build user-facing web-based tools and dashboard that allow users to browse datasets, inspect results, and understand experimental progress over time.
• Work with a SQL-backed database to store metrics, experiment metadata, and summaries, ensuring the data can be queried and accessed consistently across systems.
• Contribute to data traceability and provenance mechanisms that capture how datasets are generated, transformed, and consumed in ML workflows.
• Data Ingestion Platform (pokedex / evdc_ingest):
  • Own and extend an existing data ingestion system responsible for uploading vehicle test data to Amazon S3.
  • Improve ingestion orchestration to support:
    • Upload prioritization for small datasets
    • Deferred upload scheduling for large datasets during off-hours
    • Automatic discarding of data explicitly marked as trash
    • Persistent queueing and resumability across server restarts or failures
  • Maintain ingestion reliability under constrained network bandwidth.
  • Extend the current web interface for clarity, reliability and extendability.
  • System-level architecture decisions will be guided by the team.
• Post-Ingestion Automation, Annotation and Storage:
  • Integrate ingestion workflows with post-processor, such as:
    • Existing LLM-based automatic annotation module
    • Automating plot generation (You come back to automatically generated plots as soon as data hits S3 - imagine that!)
  • Metric computation pipelines
  • Package and deploy the annotation system as a service (e.g., EC2-based).
• Implement orchestration logic to trigger annotation jobs opportunistically when ingestion resources are idle.
• Store metrics, experiment metadata, plots and summaries in SQL-backed database layer.
• Metrics Platform & Leaderboards:
  • Implement and extend a SQL-backed metrics database using schemas defined by the team.
  • Define schemas to support:
    • Multiple projects
    • Baselines vs experimental runs
    • Historical comparisons
• Build automated pipelines to compute and register metrics after ingestion.
• Implement project-level leaderboard functionality to track:
  • Best performance per metric
  • Accepted baselines vs rejected experiments
  • Develop a web-based visualization interface to:
    • Display time-series progress
    • Visualize metric tradeoffs
    • Summarize experimental outcomes
• Data Traceability & Provenance
  • Design and implement a data provenance system for ML datasets.
  • Track:
    • Source S3 URLs
    • Post-processing operations applied to datasets
• Implement a registry of post-processing functions with support for:
  • Easy addition and removal
  • Versioning and configuration tracking
  • Generate human-readable dataset identifiers.
  • Enable lookup and inspection of dataset lineage via API and/or web interface.

Required Qualifications:

• Experience with Python for backend services, data pipelines, and automation.
• Working knowledge of SQL, including writing queries and understanding database schemas.

• Experience building web-based tools, including:
• Backend APIs (e.g., FastAPI, Flask, or similar)

• Frontend applications using React or other modern frameworks
• Familiarity with AWS and cloud-based storage or services.

• Comfortable working in Linux environments

Preferred Qualifications:

• Interest in autonomous racing and vehicle dynamics research.
• Prior internship or project experience involving data pipelines, dashboards, or analytics tools.

• Exposure to data visualization libraries, ML workflows, or experiment tracking systems.

If hired, you will enjoy the following ECLARO Benefits:

• 401k Retirement Savings Plan administered by Merrill Lynch
• Commuter Check Pretax Commuter Benefits
• Eligibility to purchase Medical, Dental & Vision Insurance through ECLARO

If interested, you may contact:
Greg Papalexis
...@eclaro.com
(646) ###-####


Equal Opportunity Employer: ECLARO values diversity and does not discriminate based on Race, Color, Religion, Sex, Sexual Orientation, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status, in compliance with all applicable laws.