Terraform
How To Wrap Your Terraform Provider for Pulumi
Thomas Rooney
September 15, 2023
In this post, we'll show you how to make a Terraform provider available on Pulumi. The article assumes some prior knowledge of Terraform and Pulumi, as well as fluency in Go. If you are new to Terraform providers, please check out our Terraform documentation.
While we provide instructions for building a Pulumi provider here, the resultant provider is not intended for production use. If you want to maintain a Pulumi provider as part of your product offering, we recommend you get in touch with us. Without a partner, providers can become a significant ongoing cost.
Why Users Are Switching From Terraform to Pulumi
Following the recent HashiCorp license change (opens in a new tab), many users are exploring Pulumi as an alternative to Terraform.
The license change came as a surprise. In response, many companies are considering alternatives to manage their infrastructure-as-code setup. Given the scale of the Terraform ecosystem, it's unlikely that the license change will lead to Terraform disappearing. However, we can expect to see some fragmentation in the market.
If you're used to Terraform, you might notice that Pulumi has fewer providers available. Currently, Pulumi offers 125 providers in their registry, while Terraform boasts an incredible 3,511.
We know the comparison isn't completely fair, though – some users take the "everything-as-code" approach to the extreme, with providers for ordering pizza (opens in a new tab), building Factorio factories (opens in a new tab), and placing blocks in Minecraft (opens in a new tab). But even accounting for hobbyist providers, it's clear that Terraform has significantly more third-party support.
So, let's look at how we can shrink that provider gap...
How Pulumi Differs From Terraform
Pulumi and Terraform are both infrastructure-as-code tools, but they differ in many ways (opens in a new tab), most importantly in the languages they support.
Terraform programs are defined in the declarative HashiCorp Configuration Language (HCL), while Pulumi allows users to create imperative programs using familiar programming languages like Python, Go, JavaScript, TypeScript, C#, and Java.
This difference has some benefits and drawbacks. With Pulumi's imperative approach, users have more control over how their infrastructure is defined and can write complex logic that isn't easily expressed in Terraform's declarative language. However, this also means that Pulumi code can be less readable than Terraform code.
Bridging Terraform and Pulumi
Pulumi provides two tools to help maintainers build bridge providers. The first is Pulumi Terraform Bridge (opens in a new tab), which creates Pulumi SDKs based on a Terraform provider schema. The second repository, Terraform Bridge Provider Boilerplate (opens in a new tab), is a template for building a new Pulumi provider based on a Terraform provider.
While creating a new provider, we'll use the Terraform Bridge Provider Boilerplate, but we'll often call functions from the Pulumi Terraform Bridge.
Pulumi Terraform Bridge is actively maintained, so bear in mind that the requirements and steps below may change with time.
How the Pulumi Terraform Bridge Works
Pulumi Terraform Bridge plays an important role during two distinct phases: design time and runtime.
During design time, Pulumi Terraform Bridge inspects a Terraform provider schema, then generates Pulumi SDKs in multiple languages.
At runtime, the bridge connects Pulumi to the underlying resource via the Terraform provider schema. This way, the Terraform provider schema continues to perform validation and calculates differences between the state in Pulumi and the resource state.
Pulumi Terraform Bridge does not use the Terraform provider binaries. Instead, it creates a Pulumi provider based only on a Terraform provider's Go modules and provider schema.
Step by Step: Creating a Terraform Bridge Provider in Pulumi
The process of creating a Pulumi provider differs slightly depending on how the Terraform provider was created. In the past, most Terraform providers were based on Terraform Plugin SDK (opens in a new tab). More recent providers are usually based on Terraform Plugin Framework (opens in a new tab).
The steps you'll follow depend on your Terraform provider. Inspect the go.mod
file in your provider to see whether it depends on github.com/hashicorp/terraform-plugin-sdk
or github.com/hashicorp/terraform-plugin-framework
.
Prerequisites
We manually installed the required tools before noticing that the Terraform Bridge Provider Boilerplate contains a Dockerfile to set up a development image. The manual process wasn't too painful, and either method should work.
The following must be available in your $PATH
:
pulumictl
(opens in a new tab)- Pulumi (opens in a new tab)
- Go 1.17 (opens in a new tab) or 1.latest
- NodeJS (opens in a new tab) 14.x – we recommend using nvm (opens in a new tab) to manage Node.js installations
- Yarn (opens in a new tab)
- TypeScript (opens in a new tab)
- Python (opens in a new tab) (called as
python3
) – for recent versions of macOS, the system-installed version is fine - .NET (opens in a new tab)
Terraform Plugin SDK to Pulumi
As an example of a Terraform provider based on Terraform Plugin SDK, we'll use this Spotify Terraform provider (opens in a new tab), a small provider for managing Spotify playlists with Terraform.
To create a new Pulumi provider, we'll start with the Terraform Bridge Provider Boilerplate (opens in a new tab).
Clone the Terraform Bridge Provider Boilerplate
Go to the Terraform Bridge Provider Boilerplate (opens in a new tab) repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.
Select your organization as the owner of the new repository (we'll use speakeasy-api
) and create a name for your Pulumi provider. In Pulumi, it is conventional to use pulumi-
followed by the resource name in lowercase as the provider name. We'll use pulumi-spotify
.
Click Create repository.
Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.
In the terminal, replace speakeasy-api
with your GitHub organization name in the code below and run it:
git clone git@github.com:speakeasy-api/pulumi-spotify.gitcd pulumi-spotify
Rename Pulumi-Specific Strings in the Boilerplate
The Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi. We need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.
The Makefile
has a prepare
command that handles some of the string replacement. In the terminal, replace speakeasy-api
with your GitHub organization name in the command below and run it:
make prepare NAME=spotify REPOSITORY=github.com/speakeasy-api/pulumi-spotify
The make
command will print the sed
commands it ran to replace the boilerplate strings in two files in the repo.
Next, we'll use sed
to replace strings in the rest of the repo:
In the terminal, replace speakeasy-api
with your GitHub organization name, then run:
find . -not \( -name '.git' -prune \) / -not -name 'Makefile' / -type f / -exec sed / -i '' 's|github.com/pulumi/pulumi-spotify|github.com/speakeasy-api/pulumi-spotify|g' {} \;
In the Makefile
, replace the ORG
variable with the name of your GitHub organization.
Finally, in the provider/resources.go
file, manually replace the values in the tfbridge.ProviderInfo
struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the GitHubOrg
to conradludgate
.
Import Your Terraform Provider
Back in the provider/resources.go
file, replace the github.com/terraform-providers/terraform-provider-spotify/spotify
import with github.com/conradludgate/terraform-provider-spotify/spotify
.
In a terminal run the following to change into the provider directory and install requirements.
cd providergo mod tidy
Fix a Dependency Version
This temporary step is a workaround related to the version of terraform-plugin-sdk
imported in the boilerplate.
During our testing, we encountered a bug in terraform-plugin-sdk
that was fixed in v2.0.0-20230710100801-03a71d0fca3d
.
In provider/go.mod
, replace replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20220824175045-450992f2f5b9
with replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20230710100801-03a71d0fca3d
. Note the difference in the version strings.
Remove Outdated make
Step
This temporary step removes a single line from the Makefile
that copies a nonexistent scripts
directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts
folder containing install-pulumi-plugin.js
, but Pulumi no longer generates these files (opens in a new tab).
In the Makefile
, remove the cp -R scripts/ bin && \
line from build_nodejs
:
Remove the Nonexistent prov.MustApplyAutoAliasing()
Function
In provider/resources.go
, remove the line prov.MustApplyAutoAliasing()
from the Provider()
function.
Build the Generator
In the terminal, run:
make tfgen
Go will build the pulumi-tfgen-spotify
binary. You can safely ignore any warnings about missing documentation. This can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide because the Pulumi boilerplate code does not include this step yet.
Build the Provider
In the terminal, run:
make provider
Go now builds the pulumi-resource-spotify
binary and outputs the same warnings as before.
Build the SDKs
In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.
In the terminal, run:
make build_sdks
You can find the generated SDKs in the new sdk
directory in your repository.
Terraform Plugin Framework to Pulumi
As we mentioned earlier, more recent Terraform plugins are based on Terraform Plugin Framework instead of Terraform Plugin SDK. The way Terraform Plugin Framework structures Go code adds a few extra steps when bridging a plugin to Pulumi.
The difference is significant enough that Pulumi Terraform Bridge includes a dedicated tool called Pulumi Bridge for Terraform Plugin Framework (opens in a new tab) in its main repository.
As with providers created using Terraform Plugin SDK, Pulumi Bridge for Terraform Plugin Framework needs to create a new Go binary that calls the Terraform plugin's new provider function. Plugins created with Terraform Plugin Framework define their new provider functions in an internal package, which means we can't import the package directly.
To work around this, we'll create a shim that imports the internal package and exposes a function to our bridge.
But we're getting ahead of ourselves. Let's look at a step-by-step example.
Airbyte Terraform Provider
For this example, we'll bridge the Airbyte Terraform provider (opens in a new tab) to Pulumi. While not the focus of this guide, it is worth mentioning that this provider was entirely generated by Speakeasy (opens in a new tab).
Clone the Terraform Bridge Provider Boilerplate
Go to the Terraform Bridge Provider Boilerplate (opens in a new tab) repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.
Select your organization as the owner of the new repository (we'll use speakeasy-api
again) and create a name for your Pulumi provider. Let's use pulumi-airbyte
.
Click Create repository.
Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.
In the terminal, replace speakeasy-api
with your GitHub organization name in the code below and run it:
git clone git@github.com:speakeasy-api/pulumi-airbyte.gitcd pulumi-airbyte
Rename Pulumi-Specific Strings in the Boilerplate
You'll remember that the Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi, so we need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.
The Makefile
has a prepare
command that handles some of the string replacement. In the terminal, replace speakeasy-api
with your GitHub organization name in the command below and run it:
make prepare NAME=airbyte REPOSITORY=github.com/speakeasy-api/pulumi-airbyte
The make
command will print the sed
commands it ran to replace the boilerplate strings in two files in the repo.
Next, we'll use sed
to replace strings in the rest of the repo.
In the terminal, replace speakeasy-api
with your GitHub organization name, then run:
find . -not \( -name '.git' -prune \) / -not -name 'Makefile' / -type f / -exec sed / -i '' 's|github.com/pulumi/pulumi-airbyte|github.com/speakeasy-api/pulumi-airbyte|g' {} \;
In the Makefile
, replace the ORG
variable with the name of your GitHub organization.
In the provider/resources.go
file, manually replace the values in the tfbridge.ProviderInfo
struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the GitHubOrg
to your airbytehq
.
Most importantly, in provider/resources.go
, replace fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg),
with fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),
:
Remember to replace speakeasy-api
with your organization name.
Remove Outdated make
Step
This temporary step removes a single line from the Makefile
that copies a nonexistent scripts
directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts
folder containing install-pulumi-plugin.js
, but Pulumi no longer generates these files (opens in a new tab).
In the Makefile
, remove the cp -R scripts/ bin && \
line from build_nodejs
:
Remove the Nonexistent ‘prov.MustApplyAutoAliasing()’ Function
In provider/resources.go
, remove the line prov.MustApplyAutoAliasing()
from the Provider()
function.
Create a Shim To Import the Internal New Provider Function
Start with a new directory called provider/shim
in your pulumi-airbyte
project:
mkdir provider/shim
Add a go.mod
file to this directory with the following contents:
Now we'll add shim.go
to this directory:
Add Shim Requirements
To have Go gather the requirements for our shim module, run the following from the root of the project:
cd provider/shimgo mod tidycd ../..
Import the New Shim Provider and the Terraform Package Framework Bridge
In provider/resources.go
, edit your imports to look like this (replace speakeasy-api
with your organization name):
Instantiate the Shimmed Provider
In provider/resources.go
, replace shimv2.NewProvider(airbyte.Provider())
with pf.ShimProvider(airbyteshim.NewProvider())
:
Add the Shim Module as a Requirement
Edit provider/go.mod
, and add github.com/airbytehq/terraform-provider-airbyte/shim v0.0.0
to the requirements.
Also in provider/go.mod
, replace github.com/airbytehq/terraform-provider-airbyte/shim
with ./shim
as shown below, to let the Go compiler look for the shim in our local repository:
Install Go Requirements
From the root of the project, run:
cd providergo mod tidycd ..
Build the Generator
In the terminal, run:
make tfgen
Go will build the pulumi-tfgen-airbyte
binary. You can safely ignore any warnings about missing documentation. The missing documentation warnings can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide as the Pulumi boilerplate code does not include this step yet.
Build the Provider
In the terminal, run:
make provider
Go now builds the pulumi-resource-airbyte
binary and outputs the same warnings as before.
Build the SDKs
In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.
In the terminal, run:
make build_sdks
You can find the generated SDKs in the new sdk
directory in your repository.
Summary
We hope this comparison of Terraform and Pulumi and our step-by-step guide to bridging a Terraform provider into Pulumi has been useful to you. You should now be able to create a Pulumi provider based on your Terraform provider.
Speakeasy can help you generate a Terraform provider based on your OpenAPI specifications. Follow our documentation to enter this exciting ecosystem.
Speakeasy is considering adding Pulumi support. Join our Slack community (opens in a new tab) to discuss this or for expert advice on Terraform providers.