Deploying Features in a Repository to a Workspace
Your Tecton Repository contains Tecton Object Definitions which define feature pipelines and other dataflows.
When you make changes to your Repository locally and apply them to a selected
Target Workspace (using the tecton apply
CLI command), Tecton does the
following:
- Tecton identifies the diff between the Target Workspace's current state vs. the newly-applied state and validates any new or updated Object Definitions
- Tecton updates the Workspace Configuration accordingly (e.g. by setting up required pipelines, spinning up online or offline table(s), modifying Feature View definitions, connecting to new Data Sources, etc.).
- The changes are applied to the
current workspace
that is selected in your Tecton CLI. (To see the selected Workspace, run
tecton workspace show
. To change the selected Workspace, runtecton workspace select your_workspace
). - Changes that have been applied become available on your Tecton instance; for example, feature pipelines run against applied features and write the results to the online and offline stores.
Making a change to a Tecton Repository​
As an example, add the file my_entity.py
, which contains an Entity
object,
to a local Tecton Repository:
# my_repo/my_entity.py
from tecton import Entity
user = Entity(name="user", join_keys=["user_id"], description="My first entity!")
It's important to declare Tecton Objects as global variables in your Python
module. When plan
or apply
commands are run, the Tecton CLI references all
Python objects instantiated in the global
scope to identify objects in the
Tecton Repository.
Previewing changes using tecton plan
​
To preview changes made to your local Tecton Repository, run tecton plan
.
These are the changes that will be applied if tecton apply
is run (the next
step).
$ tecton plan
Using workspace "my_workspace" on cluster https://my_app.tecton.ai
✅ Imported 1 Python module from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
+ Create Entity
name: user
description: My first entity!
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Pushing changes using tecton apply
​
Running tecton apply
will generate the same output as tecton plan
, along
with a final prompt to apply the changes.
$ tecton apply
Using workspace "my_workspace" on cluster https://my_app.tecton.ai
✅ Imported 1 Python module from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
+ Create Entity
name: user
description: My first entity!
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Are you sure you want to apply this plan? [y/N]> y
🎉 all done!
If you see the message "all done!", as shown above, then your changes have been pushed to to the Tecton Repository.
If your Repository contains multiple changes, tecton apply
will apply all
changes at once.
Running tecton plan
before tecton apply
is not required.
Types of Repository changes​
There are 5 types of changes that can be applied to a Repository:
+ Create
: A new object is being created for the first time.- Delete
: A previously created object is being deleted.~ Recreate
: A change to an existing Tecton Object that requires it to be recreated in the remote Tecton Repository. This often occurs when transformations are updated or dependencies change between Objects. For example, changing a Data Source definition may require any Feature Views that depend on it to be recreated and re-materialized. This is also known as a destructive update.~ Update
: An update to an existing Tecton Object's properties which doesn't require it to be recreated (e.g. changing thedescription
of a Feature View). This is also known as a non-destructive update.~ Upgrade
: No-op updates of Objects to meet the latest Tecton API version. These are sometimes observed after upgrading the Tecton SDK usingpip3
and should be considered safe.
Examples​
Below are examples of create, delete, and update changes:
$ tecton apply
Using workspace "prod"
✅ Imported 3 Python modules from the feature repository
✅ Collecting local feature declarations
✅ Performing server-side validation of feature declarations
↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
- Delete Entity
name: my_entity
owner: alice
+ Create Entity
name: my_new_entity
owner: alice
~ Update FeatureView
name: my_feature_view
owner: alice
description: -> Description of this FeatureView!
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Are you sure you want to apply this plan? [y/N]> y
🎉 all done!
Other actions related to tecton apply
​
Viewing the apply
history for a Workspace​
The tecton log
command will display a list of previously applied ids for the
remote Tecton Repository in your current Workspace.
$ tecton log
Using workspace "prod"
Apply ID: 006ad43e0000000000000107
Author: drake
Date: 2020-05-20 23:19:41.829000
Apply ID: 83a205340000000000000105
Author: rihanna
Date: 2020-05-20 18:00:01.858000
Apply ID: 56e8a66a00000000000000fd
Author: jayz
Date: 2020-05-19 15:13:35.083000
Apply ID: 4bfe16ea00000000000000f4
Author: alicakeys
Date: 2020-05-18 18:45:21.232000
Restoring a previous apply
​
Tecton stores a snapshot of your Workspace's local Tecton Repository each time
tecton apply
is run. The tecton restore
command makes it possible to
overwrite your local Tecton Repository with a previous applied version.
To restore the most recently applied version, run tecton restore
without an
apply ID:
$ tecton restore
Using workspace "prod"
This operation may remove or modify the following files:
/Users/drake/Tecton/my-git-repo/feature_repo/entities.py
/Users/drake/Tecton/my-git-repo/feature_repo/data_sources.py
/Users/drake/Tecton/my-git-repo/feature_repo/feature_views.py
Ok? [y/N]>y
To restore previous version of your local Tecton Repository, first run
tecton log
to determine which apply ID to restore, then run
tecton restore <apply ID>
.
$ tecton log
Using workspace "prod"
Apply ID: 006ad43e0000000000000107
Author: jayz
Date: 2020-05-20 23:19:41.829000
Apply ID: **83a205340000000000000105**
Author: rihanna
Date: 2020-05-20 18:00:01.858000
Apply ID: 4bfe16ea00000000000000f4
Author: alicakeys
Date: 2020-05-18 18:45:21.232000
$ tecton restore **83a205340000000000000105**
tecton restore
Using workspace "prod"
This operation may remove or modify the following files:
/Users/drake/Tecton/my-git-repo/feature_repo/entities.py
/Users/drake/Tecton/my-git-repo/feature_repo/data_sources.py
/Users/drake/Tecton/my-git-repo/feature_repo/feature_views.py
Ok? [y/N]>y
Skipping files using .tectonignore
​
Tecton supports a .tectonignore
file that can specify files or path
expressions to ignore when running plan
or apply
. It's similar to Git's
.gitignore
configuration. .tectonignore
should be declared in the feature
repo root directory.
When running tecton plan
or tecton apply
, Tecton's CLI processes all objects
in .py
files within a folder containing a .tecton
file (created by running
tecton init
). For example, consider the following repo with objects declared
in transactions_batch.py
, entities.py
, fraud_detection.py
, and
fraudulent_transactions_count.py
.
├── data_sources
│  └─ ─ transactions_batch.py
├── entities.py
├── feature_services
│  └── fraud_detection.py
└── features
└── fraudulent_transactions_count.py
Suppose everything under feature_services/
needed to be temporarily ignored.
Adding .tectonignore
to the repo root with the following glob expression will
ignore the file fraud_detection.py
altogether.
# Ignore everything under feature_services/
feature_services/*.py
# Alternatively, include nested directories under feature_services/
feature_services/**/*.py
# Alternatively, ignore a specific file
feature_services/fraud_detection.py
When running tecton plan
or tecton apply
, Tecton's CLI would see the
following file structure:
├─ ─ .tectonignore
├── data_sources
│  └── transactions_batch.py
├── entities.py
└── features
└── fraudulent_transactions_count.py
Since fraud_detection.py
is now ignored, Tecton's CLI will flag that any
objects that were defined in the file and previously applied would be deleted
upon running tecton apply
.
Cloning an existing Workspace​
Running tecton apply
within a workspace simply applies the local feature repo
to the Workspace. Suppose you would like to iterate on a live Workspace in an
experimental development Workspace using a new git branch.
- Select the existing Workspace using
tecton workspace select
:
$ tecton workspace select existing_workspace
Switched to workspace "existing_workspace".
- (Optional) Restore the existing Workspace to your local repo
$ tecton restore
- Create a new workspace
$ tecton workspace create my_workspace
Created workspace "my_workspace".
Switched to workspace "my_workspace".
You have created a new, empty workspace. Workspaces let
you create and manage an isolated feature repository.
Running "tecton plan" will compare your local repository
against the remote repository, which is initially empty.
-
Apply the current state to your new workspace.
The new workspace
my_workspace
is empty. Runtecton apply
so that the new workspace reflects the currentprod
state, which has been captured in your local Git branch.$ tecton apply
✅ Imported 11 Python modules from the feature repository
✅ Collecting local feature declarations
↓↓↓↓↓↓↓↓↓↓↓↓ Plan Start ↓↓↓↓↓↓↓↓↓↓
...
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Are you sure you want to apply this plan? [Y/n] Y
All done! -
Create a new Git branch
To develop using the existing Workspace state as a base, create a new Git branch from main in which to store changes made in your workspace:
# Create a new git branch
git checkout -b "my_workspace_branch"
Switched to new branch "my_workspace_branch"
You now have a development environment with a workspace and a Git branch that
reflect the current state of existing_workspace
.