Okko Oulasvirta

Azure, DevOps & AI Solution Architect

Introduction

Microsoft Fabric provides powerful analytics capabilities, but enterprises need reliable methods to deploy and manage resources across environments. Infrastructure as Code (IaC) has emerged as a DevOps best practice across cloud platforms, and the recently announced Microsoft Fabric Provider for Terraform extends this capability to Fabric resources.

‼️NOTE: Provider is still currently in experimental state and not recommended to production workloads. Latest version is release candidate 0.1.0-rc.1 as of 14.3.2024

The Terraform provider introduces a declarative approach where entire Fabric workspaces and related resources can be defined, versioned, and deployed with complete consistency and auditability—critical requirements for enterprise governance and compliance.

This part 1 of evaluation concentrates first on capabilities of the Terraform provider using simple DEV & TEST workspace architecture where DEV is configured with Git integration and Terraform state management is stored locally on developer machine.

For simplicity in all scenarios, Fabric workspaces already exist - they are not created with Terraform.

Prerequisites

  • Terraform tooling installed on development workstation.
  • Existing Microsoft Fabric capacity (trial capacity is not supported).
  • Existing Fabric workspaces for DEV and TEST environments.
  • Fabric account with contributor access to the workspaces.

Key Concepts

Declarative Resource Management - Terraform allows you to define the desired state of your Fabric resources in infrastructure-as-code rather than writing procedural scripts to create them.

State Management - Terraform tracks the real-world resources in its state file, allowing it to detect drift and make incremental changes.

Dependency Management - Dependencies between Fabric resources are automatically managed by Terraform’s resource graph.

Setup a Local Development Environment

In this section, we’ll configure a local Terraform environment to manage Fabric resources across multiple workspaces. This setup enables consistent deployment patterns while maintaining environment isolation through Terraform workspaces.

We’ll focus on three key components:

  1. Terraform workspace configuration for state management
  2. Fabric provider installation and authentication
  3. Environment-specific variable configuration for DEV and TEST

local-state-dev-test

Create a Terraform workspaces for DEV and TEST environments

Create a new Terraform workspace for the DEV and TEST Fabric workspace deployments in different environment stages.

terraform workspace new dev
terraform workspace new test

This will create a terraform.tfstate.d directory with dev and test subdirectories for storing the Terraform state files. Use terraform workspace list to see the available workspaces and terraform workspace show to see the current workspace. First test deployments are going to be done against development workspace, so I’ll select the Terraform dev workspace as current state management context.

terraform workspace select dev

Install Terraform Fabric Provider

Microsoft Fabric Terraform provider needs to be installed on the development workstation. Add the following Terraform provider configuration and run terraform init to download the provider plugin.

terraform {
    required_version = ">= 1.8.0"
    required_providers {
        fabric = {
            source  = "microsoft/fabric"
            version = "0.1.0-rc.1"
        }
    }
}

provider "fabric" {
    use_cli = true # Allow Azure CLI to be used for authentication.
    preview = true #  Enable preview mode to use preview features.
}

providers.tf

Verify latest available version in Microsoft Fabric Provider docs.

Add Terraform variables and values for DEV and TEST workspace deployments

Define Terraform variables for the DEV workspace in a variables.tf file and set values for the variables to customize the deployment for the DEV dev.tfvars and TEST test.tfvars workspaces.

variable "workspace_id" {
    description = "An id (guid) for targeted workspace."
    type        = string    
}

variables.tf

Workspace ID can be found in the Fabric portal URL.

workspace id can be found in the url

Values for the variables are set in the variables.dev.tfvars file.

# The DEV Fabric Workspace ID (guid can be found in the URL of the workspace)
workspace_id = "d03d9bfb-0a87-4646-9399-ceaccb437f53"

variables.dev.tfvars

and for the TEST workspace in the variables.test.tfvars file.

# The TEST Fabric Workspace ID (guid can be found in the URL of the workspace)
workspace_id = "2ac9716d-ce8d-4508-9b84-9eff009fbffa"

variables.test.tfvars

🧪 Terraform first approach - Deploy Fabric Items to DEV Workspace with Terraform

In this scenario, we start by defining Fabric resources as Terraform templates first, then deploy them to create the actual resources in the DEV workspace. This approach follows the infrastructure-as-code principle where all resources are declared and versioned in code before they exist in the environment.

We’ll define and deploy multiple Fabric resource types, demonstrating how Terraform can manage various components of your Fabric workspace:

  1. Define resource templates for each Fabric item type
  2. Apply the templates to create resources in the DEV workspace
  3. Verify the resources are created correctly

Let’s walk through creating templates for each Fabric resource type and applying them to our DEV workspace.

Fabric Lakehouse - Terraform Resource Template

To test deployment of non-existing Fabric item, create a new Terraform template file lakehouse.tf and add the following code to define a Fabric workspace and a lakehouse resource.

# Terraform template for Microsoft Fabric Lakehouse
resource "fabric_lakehouse" "lakehouse" {
    display_name        = "lakehouse"
    description         = "Managed with Terraform template lakehouse.tf"
    workspace_id        = var.workspace_id
}

# Output the lakehouse ID for use by other resources
output "lakehouse_id" {
    value       = fabric_lakehouse.lakehouse.id
    description = "The ID of the created Fabric lakehouse."
}

lakehouse.tf

Run terraform plan -var-file=variables.dev.tfvars to see what changes Terraform will make to the DEV workspace. It should report that it will create a new lakehouse resource.

terraform plan -compact-warnings -var-file=variables.dev.tfvars

Plan outputs Plan: 1 to add, 0 to change, 0 to destroy. as expected. The Lakehouse Fabric item doesn’t yet exist in the DEV workspace. Now creating it with terraform apply command.

terraform apply -var-file="variables.dev.tfvars" -auto-approve -no-color -out "dev.tfout"

fabric_lakehouse.lakehouse: Creating...
fabric_lakehouse.lakehouse: Still creating... [10s elapsed]
fabric_lakehouse.lakehouse: Creation complete after 36s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Fabric Warehouse - Terraform Resource Template

Add a new Terraform template file warehouse.tf and add the following code to define a Fabric warehouse resource.

# Terraform template for Microsoft Fabric Warehouse
resource "fabric_warehouse" "warehouse" {
    display_name        = "warehouse"
    description         = "Managed with Terraform template warehouse.tf"
    workspace_id        = var.workspace_id
}

# Output the warehouse ID for use by other resources
output "warehouse_id" {
    value       = fabric_warehouse.warehouse.id
    description = "The ID of the created Fabric warehouse."
}

warehouse.tf

Fabric Data Pipeline - Terraform Resource Template

Add a new Terraform template file data-pipeline.tf and add the following code to define a Fabric data pipeline resource.

# Terraform template for Microsoft Fabric DataPipeline
resource "fabric_data_pipeline" "datapipeline" {
    display_name        = "datapipeline"
    description         = "Managed with Terraform template data_pipeline.DataPipeline.tf."
    workspace_id        = var.workspace_id
    format              = "Default"
}

# Output the data pipeline ID for use by other resources
output "datapipeline_id" {
    value       = fabric_data_pipeline.datapipeline.id
    description = "The ID of the created Fabric data pipeline"
}

pipeline.tf

Fabric Notebook - Terraform Resource Template

Add a new Terraform template file notebook.tf and add the following code to define a Fabric notebook resource. Notebooks are key components for data analysis and transformation in Fabric, providing a code-first interface for data professionals.

# Terraform template for Microsoft Fabric Notebook
resource "fabric_notebook" "notebook" {
    display_name        = "notebook"
    description         = "Managed with Terraform template notebook.tf"
    workspace_id        = var.workspace_id
}

# Output the notebook ID for use by other resources
output "notebook_id" {
    value       = fabric_notebook.notebook.id
    description = "The ID of the created Fabric notebook."
}

notebook.tf

Fabric EventStream - Terraform Resource Template

Create a new Terraform template file eventstream.tf to define a Fabric eventstream resource. EventStreams in Fabric enable real-time data ingestion and processing, ideal for scenarios requiring continuous data analysis.

# Terraform template for Microsoft Fabric EventStream
resource "fabric_eventstream" "eventstream" {
    display_name        = "eventstream"
    description         = "Managed with Terraform template eventstream.tf"
    workspace_id        = var.workspace_id
}

# Output the eventstream ID for use by other resources
output "eventstream_id" {
    value       = fabric_eventstream.eventstream.id
    description = "The ID of the created Fabric eventstream."
}

eventstream.tf

Fabric EventHouse - Terraform Resource Template

Create a new Terraform template file eventhouse.tf to manage an eventhouse resource with Terraform. EventHouses provide storage and querying capabilities for event data captured by eventstreams.

# Terraform template for Microsoft Fabric EventHouse
resource "fabric_eventhouse" "eventhouse" {
    display_name        = "eventhouse"
    description         = "Managed with Terraform template eventhouse.tf"
    workspace_id        = var.workspace_id
}

# Output the eventhouse ID for use by other resources
output "eventhouse_id" {
    value       = fabric_eventhouse.eventhouse.id
    description = "The ID of the created Fabric eventhouse."
}

eventhouse.tf

🚀 Apply Terraform Templates to Create Fabric Resources in DEV Workspace

After defining the five new Terraform templates for Fabric resource types, we can now apply them to create the resources in the DEV workspace. Run the terraform plan command to see the changes Terraform will make to the DEV workspace.

terraform plan -var-file=variables.dev.tfvars -out "dev.tfplan"
.
.
.
Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  ~ datapipeline_id = (known after apply)
  ~ eventhouse_id   = (known after apply)
  ~ eventstream_id  = (known after apply)
  ~ notebook_id     = (known after apply)
  ~ warehouse_id    = (known after apply)

The plan output shows that Terraform will create five new resources in the DEV workspace: data pipeline, eventhouse, eventstream, notebook, and warehouse. Also it states that outputs will be updated with the new resource IDs.

Now run the terraform apply command using the plan we created. This will ensure that the changes are applied as expected and with variable values from variables.dev.tfvars at the time of the plan creation.

terraform apply "dev.tfplan"
.
.
.
fabric_warehouse.warehouse: Creating...
fabric_data_pipeline.datapipeline: Creating...
fabric_eventstream.eventstream: Creating...
fabric_notebook.notebook: Creating...
fabric_eventhouse.eventhouse: Creating...
.
.
.
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

The apply command successfully created all five new Fabric resources in the DEV workspace. We can verify the resources in the Fabric portal to ensure they were created correctly.

fabric resources created in dev workspace

🚀 Switch Terraform workspace context to TEST and deploy Resources

Now that we’ve successfully deployed Fabric resources to the DEV workspace, let’s switch the Terraform workspace context to TEST and deploy the same resources to the TEST workspace. This demonstrates how Terraform can manage resources across different environments using the same templates but with different parameter values.

terraform workspace select test

Run the terraform plan command to see the changes Terraform will make to the TEST workspace.

terraform plan -var-file=variables.test.tfvars -out "test.tfplan"
.
.
.
Plan: 6 to add, 0 to change, 0 to destroy.

The plan output shows that Terraform will create six new resources in the empty TEST workspace: lakehouse, data pipeline, eventhouse, eventstream, notebook, and warehouse. Run the terraform apply command using the plan we created to deploy the resources to the TEST workspace.

terraform apply "test.tfplan"
fabric_eventstream.eventstream: Creating...
fabric_notebook.notebook: Creating...
fabric_lakehouse.lakehouse: Creating...
fabric_data_pipeline.datapipeline: Creating...
fabric_warehouse.warehouse: Creating...
fabric_eventhouse.eventhouse: Creating...
.
.
.
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Now we have identical Fabric resources deployed to both DEV and TEST workspaces using the same Terraform templates. This demonstrates how Terraform can manage resources consistently across different environments with environment-specific configurations.

fabric resources created in test workspace

Summary

In this first part of our Fabric DevOps evaluation, we’ve successfully demonstrated how the Microsoft Fabric Terraform provider has potential to address some of the key enterprise DevOps requirements. We’ve evaluated how Terraform enables:

  • đźź© Environment isolation (EDR-01): By using Terraform workspaces to manage state separately for DEV and TEST environments. NOTE: This is a simple local state management, not yet suitable for team collaboration.
  • âś… Version control (EDR-03): By defining Fabric resources as code in Terraform templates.
  • âś… Automation (EDR-04): By deploying multiple Fabric items consistently with parameterization
  • âś… Cross-environment deployments: By deploying identical resource structures to both DEV and TEST environments while maintaining environment-specific configurations

We’ve implemented and validated the “Terraform-first” approach for six different Fabric item types: Lakehouse, Warehouse, Data Pipeline, Notebook, EventStream, and EventHouse.

Our implementation demonstrated that the Terraform provider successfully handles resource creation across environments and provides a foundation for infrastructure-as-code practices with Fabric.

‼️ NOTE, we’ve only explored local state management so far, which presents challenges for team collaboration. In the next part, we’ll address this limitation by exploring the “Fabric-first” approach where existing Fabric items are imported into Terraform state and examine more advanced state management options including remote backend configurations for shared state between multiple developers.

Encountered Issues and Workarounds

Terraform is trying to use IPv6 to connect to the registry but encountering a “network is unreachable” error.

terraform init
Initializing the backend...
Initializing provider plugins...
- Finding microsoft/fabric versions matching "0.1.0-rc.1"...
â•·
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider microsoft/fabric: could not query provider registry for
│ registry.terraform.io/microsoft/fabric: the request failed after 2 attempts, please try again later: Get
│ "https://registry.terraform.io/v1/providers/microsoft/fabric/versions": dial tcp
│ [2600:9000:278f:c800:16:1aa3:1440:93a1]:443: connect: network is unreachable
│ 
│ To see which modules are currently depending on microsoft/fabric and what versions are specified, run the following command:
│     terraform providers

Workaround is to enforce go (used by Terraform) to use IPv4 for DNS resolution:

$env:GODEBUG = "netdns=go+4"
terraform init

You May Also Enjoy

Testing how az app credential reset works

3 minute read

In application development and management, ensuring the security of your application is crucial. One such measure is routinely resetting the credentials for ...

ChatGPT Reaches One Million: What is it?

4 minute read

OpenAI’s ChatGPT has reached one million users in record time. This post explores what ChatGPT is, how it works, and what it means for enterprise AI adoption.