Terraform

How To Wrap Your Terraform Provider for Pulumi

Thomas Rooney

September 15, 2023

In this post, we'll show you how to make a Terraform provider available on Pulumi. The article assumes some prior knowledge of Terraform and Pulumi, as well as fluency in Go. If you are new to Terraform providers, please check out our Terraform documentation.

While we provide instructions for building a Pulumi provider here, the resultant provider is not intended for production use. If you want to maintain a Pulumi provider as part of your product offering, we recommend you get in touch with us. Without a partner, providers can become a significant ongoing cost.

Why Users Are Switching From Terraform to Pulumi

Following the recent HashiCorp license change (opens in a new tab), many users are exploring Pulumi as an alternative to Terraform.

The license change came as a surprise. In response, many companies are considering alternatives to manage their infrastructure-as-code setup. Given the scale of the Terraform ecosystem, it's unlikely that the license change will lead to Terraform disappearing. However, we can expect to see some fragmentation in the market.

If you're used to Terraform, you might notice that Pulumi has fewer providers available. Currently, Pulumi offers 125 providers in their registry, while Terraform boasts an incredible 3,511.

We know the comparison isn't completely fair, though – some users take the "everything-as-code" approach to the extreme, with providers for ordering pizza (opens in a new tab), building Factorio factories (opens in a new tab), and placing blocks in Minecraft (opens in a new tab). But even accounting for hobbyist providers, it's clear that Terraform has significantly more third-party support.

So, let's look at how we can shrink that provider gap...

How Pulumi Differs From Terraform

Pulumi and Terraform are both infrastructure-as-code tools, but they differ in many ways (opens in a new tab), most importantly in the languages they support.

Terraform programs are defined in the declarative HashiCorp Configuration Language (HCL), while Pulumi allows users to create imperative programs using familiar programming languages like Python, Go, JavaScript, TypeScript, C#, and Java.

This difference has some benefits and drawbacks. With Pulumi's imperative approach, users have more control over how their infrastructure is defined and can write complex logic that isn't easily expressed in Terraform's declarative language. However, this also means that Pulumi code can be less readable than Terraform code.

Bridging Terraform and Pulumi

Pulumi provides two tools to help maintainers build bridge providers. The first is Pulumi Terraform Bridge (opens in a new tab), which creates Pulumi SDKs based on a Terraform provider schema. The second repository, Terraform Bridge Provider Boilerplate (opens in a new tab), is a template for building a new Pulumi provider based on a Terraform provider.

While creating a new provider, we'll use the Terraform Bridge Provider Boilerplate, but we'll often call functions from the Pulumi Terraform Bridge.

Pulumi Terraform Bridge is actively maintained, so bear in mind that the requirements and steps below may change with time.

How the Pulumi Terraform Bridge Works

Pulumi Terraform Bridge plays an important role during two distinct phases: design time and runtime.

During design time, Pulumi Terraform Bridge inspects a Terraform provider schema, then generates Pulumi SDKs in multiple languages.

At runtime, the bridge connects Pulumi to the underlying resource via the Terraform provider schema. This way, the Terraform provider schema continues to perform validation and calculates differences between the state in Pulumi and the resource state.

Pulumi Terraform Bridge does not use the Terraform provider binaries. Instead, it creates a Pulumi provider based only on a Terraform provider's Go modules and provider schema.

Step by Step: Creating a Terraform Bridge Provider in Pulumi

The process of creating a Pulumi provider differs slightly depending on how the Terraform provider was created. In the past, most Terraform providers were based on Terraform Plugin SDK (opens in a new tab). More recent providers are usually based on Terraform Plugin Framework (opens in a new tab).

The steps you'll follow depend on your Terraform provider. Inspect the go.mod file in your provider to see whether it depends on github.com/hashicorp/terraform-plugin-sdk or github.com/hashicorp/terraform-plugin-framework.

Prerequisites

We manually installed the required tools before noticing that the Terraform Bridge Provider Boilerplate contains a Dockerfile to set up a development image. The manual process wasn't too painful, and either method should work.

The following must be available in your $PATH:

• pulumictl (opens in a new tab)
• Pulumi (opens in a new tab)
• Go 1.17 (opens in a new tab) or 1.latest
• NodeJS (opens in a new tab) 14.x – we recommend using nvm (opens in a new tab) to manage Node.js installations
• Yarn (opens in a new tab)
• TypeScript (opens in a new tab)
• Python (opens in a new tab) (called as python3) – for recent versions of macOS, the system-installed version is fine
• .NET (opens in a new tab)

Terraform Plugin SDK to Pulumi

As an example of a Terraform provider based on Terraform Plugin SDK, we'll use this Spotify Terraform provider (opens in a new tab), a small provider for managing Spotify playlists with Terraform.

To create a new Pulumi provider, we'll start with the Terraform Bridge Provider Boilerplate (opens in a new tab).

Clone the Terraform Bridge Provider Boilerplate

Go to the Terraform Bridge Provider Boilerplate (opens in a new tab) repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.

Select your organization as the owner of the new repository (we'll use speakeasy-api) and create a name for your Pulumi provider. In Pulumi, it is conventional to use pulumi- followed by the resource name in lowercase as the provider name. We'll use pulumi-spotify.

Click Create repository.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace speakeasy-api with your GitHub organization name in the code below and run it:

Rename Pulumi-Specific Strings in the Boilerplate

The Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi. We need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The Makefile has a prepare command that handles some of the string replacement. In the terminal, replace speakeasy-api with your GitHub organization name in the command below and run it:

The make command will print the sed commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use sed to replace strings in the rest of the repo:

In the terminal, replace speakeasy-api with your GitHub organization name, then run:

In the Makefile, replace the ORG variable with the name of your GitHub organization.

Finally, in the provider/resources.go file, manually replace the values in the tfbridge.ProviderInfo struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the GitHubOrg to conradludgate.

Import Your Terraform Provider

Back in the provider/resources.go file, replace the github.com/terraform-providers/terraform-provider-spotify/spotify import with github.com/conradludgate/terraform-provider-spotify/spotify.

In a terminal run the following to change into the provider directory and install requirements.

Fix a Dependency Version

This temporary step is a workaround related to the version of terraform-plugin-sdk imported in the boilerplate.

During our testing, we encountered a bug in terraform-plugin-sdk that was fixed in v2.0.0-20230710100801-03a71d0fca3d.

In provider/go.mod, replace replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20220824175045-450992f2f5b9 with replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20230710100801-03a71d0fca3d. Note the difference in the version strings.

Remove Outdated make Step

This temporary step removes a single line from the Makefile that copies a nonexistent scripts directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts folder containing install-pulumi-plugin.js, but Pulumi no longer generates these files (opens in a new tab).

In the Makefile, remove the cp -R scripts/ bin && \ line from build_nodejs:

Remove the Nonexistent prov.MustApplyAutoAliasing() Function

In provider/resources.go, remove the line prov.MustApplyAutoAliasing() from the Provider() function.

Build the Generator

In the terminal, run:

Go will build the pulumi-tfgen-spotify binary. You can safely ignore any warnings about missing documentation. This can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide because the Pulumi boilerplate code does not include this step yet.

Build the Provider

In the terminal, run:

Go now builds the pulumi-resource-spotify binary and outputs the same warnings as before.

Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

You can find the generated SDKs in the new sdk directory in your repository.

Terraform Plugin Framework to Pulumi

As we mentioned earlier, more recent Terraform plugins are based on Terraform Plugin Framework instead of Terraform Plugin SDK. The way Terraform Plugin Framework structures Go code adds a few extra steps when bridging a plugin to Pulumi.

The difference is significant enough that Pulumi Terraform Bridge includes a dedicated tool called Pulumi Bridge for Terraform Plugin Framework (opens in a new tab) in its main repository.

As with providers created using Terraform Plugin SDK, Pulumi Bridge for Terraform Plugin Framework needs to create a new Go binary that calls the Terraform plugin's new provider function. Plugins created with Terraform Plugin Framework define their new provider functions in an internal package, which means we can't import the package directly.

To work around this, we'll create a shim that imports the internal package and exposes a function to our bridge.

But we're getting ahead of ourselves. Let's look at a step-by-step example.

Airbyte Terraform Provider

For this example, we'll bridge the Airbyte Terraform provider (opens in a new tab) to Pulumi. While not the focus of this guide, it is worth mentioning that this provider was entirely generated by Speakeasy (opens in a new tab).

Clone the Terraform Bridge Provider Boilerplate

Go to the Terraform Bridge Provider Boilerplate (opens in a new tab) repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.

Select your organization as the owner of the new repository (we'll use speakeasy-api again) and create a name for your Pulumi provider. Let's use pulumi-airbyte.

Click Create repository.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace speakeasy-api with your GitHub organization name in the code below and run it:

Rename Pulumi-Specific Strings in the Boilerplate

You'll remember that the Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi, so we need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The Makefile has a prepare command that handles some of the string replacement. In the terminal, replace speakeasy-api with your GitHub organization name in the command below and run it:

The make command will print the sed commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use sed to replace strings in the rest of the repo.

In the terminal, replace speakeasy-api with your GitHub organization name, then run:

In the Makefile, replace the ORG variable with the name of your GitHub organization.

In the provider/resources.go file, manually replace the values in the tfbridge.ProviderInfo struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the GitHubOrg to your airbytehq.

Most importantly, in provider/resources.go, replace fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg), with fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),:

Remember to replace speakeasy-api with your organization name.

Remove Outdated make Step

This temporary step removes a single line from the Makefile that copies a nonexistent scripts directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts folder containing install-pulumi-plugin.js, but Pulumi no longer generates these files (opens in a new tab).

In the Makefile, remove the cp -R scripts/ bin && \ line from build_nodejs:

Remove the Nonexistent ‘prov.MustApplyAutoAliasing()’ Function

In provider/resources.go, remove the line prov.MustApplyAutoAliasing() from the Provider() function.

Create a Shim To Import the Internal New Provider Function

Start with a new directory called provider/shim in your pulumi-airbyte project:

Add a go.mod file to this directory with the following contents:

Now we'll add shim.go to this directory:

Add Shim Requirements

To have Go gather the requirements for our shim module, run the following from the root of the project:

Import the New Shim Provider and the Terraform Package Framework Bridge

In provider/resources.go, edit your imports to look like this (replace speakeasy-api with your organization name):

Instantiate the Shimmed Provider

In provider/resources.go, replace shimv2.NewProvider(airbyte.Provider()) with pf.ShimProvider(airbyteshim.NewProvider()):

Add the Shim Module as a Requirement

Edit provider/go.mod, and add github.com/airbytehq/terraform-provider-airbyte/shim v0.0.0 to the requirements.

Also in provider/go.mod, replace github.com/airbytehq/terraform-provider-airbyte/shim with ./shim as shown below, to let the Go compiler look for the shim in our local repository:

Install Go Requirements

From the root of the project, run:

Build the Generator

In the terminal, run:

Go will build the pulumi-tfgen-airbyte binary. You can safely ignore any warnings about missing documentation. The missing documentation warnings can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide as the Pulumi boilerplate code does not include this step yet.

Build the Provider

In the terminal, run:

Go now builds the pulumi-resource-airbyte binary and outputs the same warnings as before.

Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

You can find the generated SDKs in the new sdk directory in your repository.

Summary

We hope this comparison of Terraform and Pulumi and our step-by-step guide to bridging a Terraform provider into Pulumi has been useful to you. You should now be able to create a Pulumi provider based on your Terraform provider.

Speakeasy can help you generate a Terraform provider based on your OpenAPI specifications. Follow our documentation to enter this exciting ecosystem.

Speakeasy is considering adding Pulumi support. Join our Slack community (opens in a new tab) to discuss this or for expert advice on Terraform providers.