Overview
A leading open-source intelligence firm is seeking a Senior Data Engineer.
The firm is a data and intelligence company specializing in digital-asset analytics for exchanges, financial institutions, regulators, and blockchain ecosystems. Technology and expertise provide clarity across crypto markets, tracking blockchain transactions, liquidity movements, and illicit finance, helping clients identify risk, enhance transparency, and improve decision-making. The firm’s infrastructure fuses structured and unstructured data from blockchains, exchanges, social networks, and financial markets.
The result is a powerful analytics engine that supports ecosystem monitoring, market surveillance, and counter–illicit finance intelligence across digital-asset networks. The firm operates as a fast-paced, nimble, global, and remote technology company and lleverage an asynchronous‑first workflow, try to minimize time spent on meetings, and believe in open debate and logic over authority. Work alongside some of the sharpest minds in the world, including intelligence analysts, defense veterans, data engineers, quant researchers, linguists, and more.
Domain Expertise
- Blockchain Data: Direct, hands-on experience treating ledger data as standard data structures. You have pulled from RPC endpoints, archive nodes, or indexers; decoded logs and ABIs; managed chain reorgs and protocol edge cases; and applied UTXO vs. account-based models in production environments.
- NLP & LLMs: Proven track record shipping production-grade text processing workflows—including extraction, classification, embeddings, vector search, or LLM-in-the-loop enrichment. You possess a clear, practical understanding of model evaluation, cost optimization, and failure mode mitigation.
Key Responsibilities
- Design, build, and operate end-to-end batch and incremental ETL/ELT pipelines, turning raw data into production-grade APIs, dashboards, and automated alerts.
- High-throughput data acquisition from diverse sources: relational databases, third-party and vendor APIs, flat files, blockchain nodes and indexers, and social and web text at scale.
- NLP and LLM-assisted processing over unstructured text — entity and claim extraction, classification, deduplication, and enrichment feeding downstream risk models.
- Model and query data across relational, object, and graph (Neo4j) stores, choosing the right one for the access pattern rather than defaulting to a favorite.
- Provenance and reproducibility: raw captures immutable, datasets versioned, and any published finding reproducible as of the date it was made.
- Data quality as a first-class deliverable: validation, lineage, reconciliation, and freshness monitoring.
- Partner with the Head of R&D, engineers, data scientists, and the investigations team to translate intelligence requirements into scalable client-facing products.
- Contribute to technical design and architecture in a written, argued design process.
Skills, Knowledge & Experience
- Advanced Python and SQL.
- Proven experience building and operating production ETL/ELT pipelines under an orchestrator (Airflow, Dagster, Prefect, Step Functions, or equivalent), including backfills, idempotency, retries, schema drift, and on-call for your own pipelines.
- Hands-on with PostgreSQL/RDS and object storage (S3).
- Serving data to consumers: building and versioning APIs (Kong OSS, FastAPI, or similar) and/or feeding dashboards.
- Docker, Git, CI/CD (GitHub Actions, AWS CodeBuild/CodePipeline), and IaC (Terraform, CDK, or equivalent).
Desirable
- Streaming and near-real-time architectures (Kafka, AWS SQS/SNS).
- Document stores (MongoDB) and analytical engines or warehouses (ClickHouse, DuckDB, Snowflake, Athena).
- Graph data modeling and querying (Neo4j/Cypher or comparable).
- Transformation and data-quality tooling (dbt, Great Expectations, Soda, OpenLineage).
- Vector stores (pgvector, Qdrant, OpenSearch).
- Go; Kubernetes.
- Domain:
- Blockchain ledger data, smart contracts, or web3 languages (Solidity, Vyper, Go, Bitcoin Script)
- NLP techniques, LLM integrations, or model feature engineering
- Financial services, trading venues, or regulatory frameworks (SEC, CFTC, FinCEN)
- OSINT, dark web, or social platform data collection
Additional Information & Benefits
- The firm operates in a fully remote, high-trust environment and prioritize impact and delivered results over hours logged.
- The firm offer competitive compensation and healthcare stipend.
- The firm invests in people through childcare stipends and dedicated employee assistance resources.
- Join a thought leader at the intersection of national security and digital assets, working with cutting-edge tech that defines the field.
