tecton.declarative.SnowflakeConfig

class tecton.declarative.SnowflakeConfig(*, database, schema, warehouse=None, url=None, role=None, table=None, query=None, timestamp_field=None, post_processor=None, data_delay=datetime.timedelta(0))

Configuration used to reference a Snowflake table or query.

The SnowflakeConfig class is used to create a reference to a Snowflake table. You can also create a reference to a query on one or more tables, which will be registered in Tecton in a similar way as a view is registered in other data systems.

This class used as an input to a BatchSource’s parameter batch_config. This class is not a Tecton Object: it is a grouping of parameters. Declaring this class alone will not register a data source. Instead, declare as part of BatchSource that takes this configuration class instance as a parameter.

Methods

__init__

Instantiates a new SnowflakeConfig.
__init__(*, database, schema, warehouse=None, url=None, role=None, table=None, query=None, timestamp_field=None, post_processor=None, data_delay=datetime.timedelta(0))

Instantiates a new SnowflakeConfig. One of table and query should be specified when creating this file.

Parameters

  • database (str) – The Snowflake database for this Data source.

  • schema (str) – The Snowflake schema for this Data source.

  • warehouse (Optional[str]) – The Snowflake warehouse for this Data source.

  • url (Optional[str]) – The connection URL to Snowflake, which contains account information (e.g. https://xy12345.eu-west-1.snowflakecomputing.com).

  • role (Optional[str]) – The Snowflake role that should be used for this Data source.

  • table (Optional[str]) – The table for this Data source. Only one of table and query must be specified.

  • query (Optional[str]) – The query for this Data source. Only one of table and query must be specified. This parameter is not supported in Tecton on Snowflake.

  • timestamp_field (Optional[str]) – The timestamp column in this data source that should be used by FilteredSource to filter data from this source, before any feature view transformations are applied. Only required if this source is used with FilteredSource.

  • post_processor (Optional[Callable]) – Python user defined function f(DataFrame) -> DataFrame that takes in raw PySpark data source DataFrame and translates it to the DataFrame to be consumed by the Feature View. This parameter is not supported in Tecton on Snowflake.

  • data_delay (timedelta) – By default, incremental materialization jobs run immediately at the end of the batch schedule period. This parameter configures how long they wait after the end of the period before starting, typically to ensure that all data has landed. For example, if a feature view has a batch_schedule of 1 day and one of the data source inputs has data_delay=timedelta(hours=1) set, then incremental materialization jobs will run at 01:00 UTC.

Returns

A SnowflakeConfig class instance.

Example of a SnowflakeConfig declaration:

from tecton import SnowflakeConfig, BatchSource

# Declare SnowflakeConfig instance object that can be used as an argument in BatchSource
snowflake_ds_config = SnowflakeConfig(
                                  url="https://<your-cluster>.eu-west-1.snowflakecomputing.com/",
                                  database="CLICK_STREAM_DB",
                                  schema="CLICK_STREAM_SCHEMA",
                                  warehouse="COMPUTE_WH",
                                  table="CLICK_STREAM_FEATURES",
                                  query="SELECT timestamp as ts, created, user_id, clicks, click_rate"
                                         "FROM CLICK_STREAM_DB.CLICK_STREAM_FEATURES")

# Use in the BatchSource
snowflake_ds = BatchSource(name="click_stream_snowflake_ds",
                               batch_config=snowflake_ds_config)

Attributes

data_delay