Our Stack.
Our team has deep technical experience with the key technologies advancing the utilization of, and the value generated by, data and AI.
While we provide an example tech stack below to help prospective clients understand our technical orientation, our team’s technical experience is in no way limited to it. Our engineers are comfortable and capable of working with a wide breadth of modern data and AI technologies.
Example Reference Architecture
Our default tech stack is composed of five complementary technologies that together form the basis of a general-purpose analytical data platform. This analytical data platform architecture is one we frequently use when working with our fractionalized engineering clients, and it provides all of the core capabilities needed to begin driving value from data and AI.
Snowflake
At the heart of our default tech stack is Snowflake, an increasingly feature-rich Data-Platform-as-a-Service (DPaaS).
At Snowflake’s core is an industry-leading analytical data store and distributed compute engine that powers both traditional analytical data and AI/ML workloads.
While Snowflake continues to expand its catalog of data platform features and capabilities, there are other complementary technologies we feel are still best-in-class.
dbt
dbt is the unrivaled technology for managing the T in ELT (Extract-Load-Transform), and brings long-standing software engineering best practices to SQL transform logic.
dbt has many flavors (i.e., open-source, cloud-managed, and Snowflake-native), and we’ve worked with all of them.
Cube
Cube began as an excellent source-agnostic, destination-agnostic, semantic layer for ensuring that all consumers of analytical data received the exact same data, but has recently become a leader in Agentic Analytics as well.
Airbyte
Airbyte is a versatile data ingestion technology with over 600 connectors, and if it doesn't have a connector you need, you can build one using its open-source SDK. Where dbt handles the T in ELT (Extract-Load-Transform), Airbyte handles the EL.
Airflow
Airflow is a long-standing workhorse in the data engineering world, used for general-purpose task orchestration. While there have been challengers over the years, Airflow remains the go-to choice for this capability.
Our Use of Artificial Intelligence
At Lucid Data, we want to leverage LLM-based AI to maximize the productivity upside, while protecting ourselves and our clients from real downside risks.
Research & Design
We actively use AI during research and design. This can help ensure we're not overlooking the best approach to solving a problem. Domain expertise is still required here to actively shape detailed requirements and separate the wheat from the chaff in AI responses.
Development
We're also increasingly using AI during development to boost productivity, but we pride ourselves on the quality of our work, so we ensure the quality that we and our clients expect, through expert prompting and thorough code reviews.
Direct Integration
Having LLM-based AI directly manage technical and business functions is extremely risky, and the technical integrations for doing so are not yet battle-tested. While we’re staying apprised of advancements here, we still consider this to be entirely research and experimentation.
Communication
While we are increasingly using LLM-based AI systems, even to check for spelling and grammar mistakes in our communications, our commitment to our clients is that our communications are our own and that none of our communications are AI-generated.