Fivetran, provider of modern data integration, extends Fivetran Transformations for dbt Core with integrated scheduling and data lineage graphs.
Support for dbt Core from dbt Labs, an open source transformation tool widely used in the data analytics community, has been available since September 2020. The current extension means a stronger integration of the open source framework dbt Core. It delivers new capabilities that help organizations simplify the complexity of the modern data stack, reduce costs through Extract-Load-Transform (ELT) automation, and accelerate data-driven decisions, according to the press release.
Transformations are an essential stage in the ELT process: raw data is processed into analysis-ready data sets, which can then be used in downstream data analytics workflows – from simple reporting to data science. Without an effective and reliable method of transformation, companies are unable to transform raw data into an analysis-ready form.
In the State of Data Engineers 2021 survey, conducted by Dimensional Research on behalf of Fivetran in March 2021, nearly half of organizations said critical data was unusable for decision-making. The study also found that 68 percent of data engineers lack the time to derive value from existing data. With Fivetran Transformations, Fivetran wants to reduce the complexity of transformations by automating the process.
By adding integrated scheduling, users can schedule their dbt core models to run automatically after syncing with a Fivetran connector. This reduces data latency and increases the speed of the end-to-end ELT pipeline.
The dbt community also experiences an additional acceleration of their analytics through the use of preconfigured data models. These are packaged SQL scripts for common data source connectors that can be run in dbt to create new reports quickly and without further data engineering.
Data lineage graphs can be used to visualize dbt Core data models so users can better track and manage their end-to-end data pipelines. Data analysts no longer have to go through the SQL code to see the relationships between the models. Data engineers get a visual representation of data movements during the transformation process. The graphs are also easy to understand and can be shared across the organization for better collaboration with data analysts and other business users.