Connecting CI/CD

Overview

We recommend creating a new CI/CD pipeline to automatically run tecton plan and tecton apply upon changes to your Tecton feature repo.

Example: Tecton Plan

For example, to run a tecton plan and tecton apply in GitHub actions, you can add the following file to the .github/workflows directory underneath your Github repository, assuming that your features repository starts at the base of your Github repository. Note that this example uses Github environments:

name: Tecton Feature Repo CI/CD

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      planid: ${{ steps.planid.outputs.planid }}
    env:
      TECTON_API_KEY: ${{ secrets.TECTON_API_KEY }}
      API_SERVICE: https://app.tecton.ai/api
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: "3.9"
          cache: "pip" # caching pip dependencies
      - run: pip install -r requirements.txt

      - name: Run tecton plan
        run: tecton plan | tee output.log

      - name: Extract plan output
        id: planid
        run: |
          PLANID=$(cat output.log | perl -wne  '/Generated plan ID is ([0-9a-f]+)/i and print $1')
          echo "planid=$PLANID" >> "$GITHUB_OUTPUT"

  apply:
    runs-on: ubuntu-latest
    needs: plan
    environment: prod-workspace
    steps:
      - env:
          PLANID: ${{needs.plan.outputs.planid}}
        run: echo "$PLANID"

Your requirements.txt file can look something like the below:

tecton[pyspark]~=0.7.3

To authenticate your Github Action, you'll need to create a Service Account to obtain a Tecton API key, and assign the Service Account the Editor role for the appropriate workspace:

1. Create the Service Account to obtain the API key.

tecton service-account create \
      --name "cicd-service-account" \
      --description "A cicd example"

Output:

Save this API Key - you will not be able to get it again.
API Key:            <Your-api-key>
Service Account ID: <Your-Service-Account-Id>

2. Assign the Editor role for the workspace to the Service Account.

tecton access-control assign-role --role editor \
      --workspace <Your-workspace> \
      --service-account <Your-Service-Account-Id>

Output:

Successfully updated role.

Validate Plan with JSON output

You can output a JSON version of a to-be-applied diff using the --json-out flag.

tecton plan --json-out <path>

This can be useful in a CI/CD pipeline to prevent applying unintended changes by running a custom script on the output.

Example json file output:

{
  "objectDiffs": [
    {
      "transitionType": "DELETE",
      "objectMetadata": {
        "name": "transaction_user_has_good_credit",
        "objectType": "FEATURE_VIEW",
        "owner": "john@doe.com",
        "description": "Whether the user had a good credit score (over 670) as of the time of a transaction."
      }
    },
    {
      "transitionType": "RECREATE",
      "objectMetadata": {
        "name": "continuous_feature_service",
        "objectType": "FEATURE_SERVICE",
        "owner": "john@doe.com",
        "description": "A Feature Service providing continuous features."
      }
    }
  ]
}

See Types of Repository Changes doc to help understand the plan output.

Apply Generated Plan

When a plan is successfully generated with tecton plan, an ID for that plan is printed to the console after the plan contents.

...
↑↑↑↑↑↑↑↑↑↑↑↑ Plan End ↑↑↑↑↑↑↑↑↑↑↑↑
Generated plan ID is a25e9516ebde475690ef3806e1f12e1e

If tecton plan was run with the --json-out flag, the plan ID is also included as a field in the JSON file:

{
  "objectDiffs": [
    ...
  ],
  "planId": "a25e9516ebde475690ef3806e1f12e1e"
}

After this plan is approved through your team's workflow (whether automated or manual), you can directly apply the plan by passing the plan ID through the --plan-id parameter:

tecton apply --plan-id=a25e9516ebde475690ef3806e1f12e1e

This will apply the plan directly without recomputing a new plan.

If any changes have been made to the feature repo since the plan was generated (i.e. someone ran tecton apply), then you will get an error and must generate a new plan on top of the current repo state.

Protecting Critical Objects from Destruction

Your repo may have critical Tecton objects that you would like to prevent from being destroyed, for example a large feature view which would be costly to rematerialize. It is possible that future state updates may accidentally delete the feature view, or trigger a destructive update (e.g. if a data source for the feature view is updated).

To protect your feature view and other critical objects from unintentionally being destroyed or recreated, you can set the prevent_destroy parameter for Feature Views or Feature Services.

For example, this Feature View is protected with the prevent_destroy parameter:

Spark

@batch_feature_view(
    sources=[FilteredSource(transactions_batch)],
    entities=[user],
    mode="spark_sql",
    online=True,
    offline=True,
    feature_start_time=datetime(2021, 5, 20),
    batch_schedule=timedelta(days=1),
    ttl=timedelta(days=30),
    description="Last user transaction amount (batch calculated)",
    prevent_destroy=True,
)
def critical_feature_view(transactions):
    pass

Snowflake

@batch_feature_view(
    sources=[FilteredSource(transactions_batch)],
    entities=[user],
    mode="snowflake_sql",
    online=True,
    offline=True,
    feature_start_time=datetime(2021, 5, 20),
    batch_schedule=timedelta(days=1),
    ttl=timedelta(days=30),
    description="Last user transaction amount (batch calculated)",
    prevent_destroy=True,
)
def critical_feature_view(transactions):
    pass

Note that the prevent_destroy has no effect in development workspaces, as there are no processes to protect from destruction.

Destroying protected objects

If at some point in the future, you want to destroy or recreate this object, this must be done in 2 steps:

1. Remove the prevent_destroy parameter from the object or set it to False, and apply this repo update. Do not combine this change with any other changes.
2. The object is unprotected at this point, so you can apply any destructive updates as you normally would.

Attempting to do the 2 steps above in a single step and running apply will return an error.