Senior Backend Engineer - Query Compiler

Narrative I/O (Headquarters: New York, NY)

Scala Spark Apache Iceberg Apache Calcite Cats Cats-Effect Http4s FS2 Doobie Deequ Axolotl BentoML HuggingFace Transformers Functional Programming Query Compiler Query Execution Distributed Systems Big Data AWS Terraform Kubernetes SQL NQL Data Collaboration Data Marketplace Backend Remote Back End Engineer Data Engineer

Narrative I/O, founded in 2016, provides a data collaboration platform focused on simplicity and ease of use. We operate a data marketplace featuring automatic data standardization, the Narrative Query Language (NQL) for flexible data access, and row-level access/pricing controls for data providers. Our team is remote-first, with engineers across North America and Europe.

We are seeking an experienced Senior Backend Engineer to focus on our NQL query compiler and backend systems. NQL is our SQL dialect with privacy and collaboration-specific extensions, compiling down for execution where data resides (e.g., Snowflake).

What You Will Do:
*   Maintain and improve our incremental view maintenance engine.
*   Update the NQL compiler to track row-level metadata for billing, governance, provenance, and lineage.
*   Build a function catalogue system for user-defined UDFs/UDAFs.
*   Develop advanced access rules and privacy controls (row-level permissions, usage restrictions, data minimization).
*   Iterate on our forecasting system for data volume/quality insights.
*   Optimize query planning, batching, and execution.
*   Perform general compiler maintenance (error messages, performance, correctness).
*   Integrate features like autocomplete and validation into our NQL editor.
*   Iterate on the structured representation of NQL for our UI.
*   Work on related systems like our API and job queue.
*   Develop a deep understanding of the business to evaluate tradeoffs and communicate technical choices.

Technical Stack:
*   Backend: Scala, Spark, Apache Iceberg, Apache Calcite, Cats, Cats-Effect, Http4s, FS2, Doobie, Deequ, Axolotl, BentoML, HuggingFace Transformers. We embrace functional programming pragmatically.
*   Operations: AWS (Fargate, EMR, RDS, Lambda, S3, ECR, ECS), Terraform, Docker, Datadog. Starting to use Kubernetes (EKS).
*   Frontend (for context): Typescript, VueJS, Nuxt, Vite.

Ideal Candidate:
*   At least one year working directly on a query compiler, engine, or closely related problem (Apache Calcite experience is ideal).
*   Solid grasp of the query processing domain.
*   Experience with large datasets and distributed query engines (e.g., Apache Spark). We ingest > 20 TiB daily.
*   Proficiency in a typed functional language (Scala, F#, Haskell) OR significant experience in Java/C# with interest in Scala/FP.
*   Experience operating in a cloud environment (AWS preferred). Kubernetes experience is a plus.
*   Willingness to contribute across the stack and operate effectively in a flat organization.
*   Ability to lead design documentation and interface with technical/non-technical members.
*   Ideally located within +/- 3 hours of US Eastern Time (EST).