How-to-Guide: Import your Existing Google Cloud Infrastructure to be managed with Terraform

Laptop Keyboard
Technology by nateserk

How-to-Guide: Import your Existing Google Cloud Infrastructure to be managed with Terraform

I initially spun up Google Cloud Compute Engine instance through Bitnami. It was working fine and suited my use-case back then. Now I was thinking more about spinning up the same setup of infrastructure in the automated fashion and doing it through Google Console/Dashboard UI Web interface wouldn’t be efficient anymore. Therefore, I decided to import my existing infrastructure on Google Cloud to be managed with Terraform. This way I can make a change, keep track of the changes made to my infrastructure into the source control (i.e. Github), and automate spinning up a new server with the exact same setup.

In this guide, I’m going to show you how to import the existing Google-Cloud compute engine to be managed with Terraform.

Why Terraform?

In short, Terraform is a powerful tool to manage infrastructure as code (IaC) the same way you manage your application code. Some benefits are the following:

  • Keep track of the changes to infrastructure with a source control (i.e github).
  • Automate your workflow when spinning up a new instance.

Prerequisite

  • Install Terraform version 0.12.x or higher
  • Install GCloud CLI
  • Install Google Auth Library (for grant access to a command line).
  • Understand and have knowledge of basic Terraform commands (init, plan, and apply)
  • Your project with Google Cloud Engine Instance
  • Google Cloud Storage for storing Terraform State

1st Setup Terraform scripts

I created these 2 Terraform scripts and replace placeholder values with yours.
You can find these values from Google Compute Engine Dashboard under VM instances section in your project.

  • variables.tf
  • resources.tf

variables.tf

Think of this as defining Hardware/OS spec for your instance.
Define the following variables in variables.tf.
These variables define which zone/region your instance is going to live and type of machine and image (OS) they are going to be.

variable "zone" { 
   type = string 
   default = "YOUR_AVAILABILITY_ZONE" 
} 

variable "region" {
   type = string
   default = "YOUR_REGION" 
} 
variable "image" {
   default = "" 
} 

variable "machine_type" { 
   default = "YOUR_MACHINE_TYPE" 
} 

variable disable-legacy-endpoints { 
   default = null 
} 
variable labels { 
   default = {} 
}

resources.tf

Think of this as defining a boot sequence, initialization and machine details for your instance.
The important parts are provider and resource. Provider tells Terraform which cloud provider to use (i.e. AWS, Azue, Google Cloud) and resource tells Terraform which type of resource it is referring to (i.e. Google Compute Engine, Google SQL, etc).

// Required variables from each project
variable project_name { default = "YOUR_PROOJECT_NAME" }
variable virtual_machine_name { default = "YOUR_INSTANCE_NAME" }
variable tags {}
variable instance_config {}

// Configure the Google Cloud provider
provider "google" {
    project     = "${var.project_name}"
    region      = "${var.region}"
}

// Google Compute Engine Instance Resource
resource "google_compute_instance" "YOUR_RESOURCE_NAME" {
  name = "${var.virtual_machine_name}"
  zone = "${var.zone}"
  project = "${var.project_name}"
  machine_type = "${var.machine_type}"
  tags = "${var.tags}"
  labels = "${var.labels}"

  metadata = {
    "status-uptime-deadline" = "420"
    "status-variable-path"   = "status"
    "disable-legacy-endpoints" = "${var.disable-legacy-endpoints}"
  }

  network_interface {
  }

}

3rd Initialize your Terraform project

Run Terraform initialization where resouces.tf is located.

terraform init

You should see the following message:

Initializing the backend...
Initializing provider plugins...

- Checking for available provider plugins...
- Downloading plugin for provider "google" (hashicorp/google) 3.17.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 3.17"


Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

4th Import Google Compute Engine Infrastructure

NOTE: The next 2 steps are going to be quite tedious because you are chasing what being missing.

Run the following command (replace YOUR_RESOURCE_NAME and YOUR_PROJECT_ID with your own values).

terraform import google_compute_instance.[YOUR_RESOURCE_NAME] [YOUR_PROJECT_ID]

You may get some errors but that’s ok. It means it recognizes you are missing some variables or required values.

For example,

google_compute_instance.YOUR_RESOURCE_NAME: Importing from ID "YOUR_PROJECT_ID"...

Error: Cannot determine zone: set in this resource, or set provider-level zone.

You’ll need to add zone inside provider block.

Once you fix that error, re-run Terraform import command again

You’re probably going to get this. This is a good news because the import is prepared.
Not-so-bad-news is there are still errors and missing fields. Let’s go to the next step.

google_compute_instance.YOUR_RESOURCE_NAME: Importing from ID "YOUR_PROJECT_ID"...

google_compute_instance.YOUR_RESOURCE_NAME: Import prepared!

  Prepared google_compute_instance for import

google_compute_instance.YOUR_RESOURCE_NAME: Refreshing state... [id=YOUR_PROJECT_ID]

Error: Cannot import non-existent remote object
While attempting to import an existing object to
google_compute_instance.YOUR_RESOURCE_NAME, the provider detected that no
object exists with the given id. Only pre-existing objects can be imported;
check that the id is correct and that it is associated with the provider's
configured region or endpoint, or use "terraform apply" to create a new remote
object for this resource.

5th Gather all the missing pieces

Run Terraform plan command:

terraform plan

Now it should tell you which require fields or variables are missing.
For example,

Error: "boot_disk": required field is not set
  on resources.tf line 16, in resource "google_compute_instance" "YOUR_RESOURCE_NAME":
  16: resource "google_compute_instance" "YOUR_RESOURCE_NAME" {

This means you need to add boot_disk section to resource “google_compute_instance” “YOUR_RESOURCE_NAME” block in resources.tf.

Let’s add boot_disk to resouces.tf and re-run terraform plan again
You should see something similar to this:

Error: "boot_disk.0.disk_encryption_key_raw": one of `boot_disk.0.auto_delete,boot_disk.0.device_name,boot_disk.0.disk_encryption_key_raw,boot_disk.0.initialize_params,boot_disk.0.kms_key_self_link,boot_disk.0.mode,boot_disk.0.source` must be specified

  on resources.tf line 16, in resource "google_compute_instance" "YOUR_RESOURCE_NAME":
  16: resource "google_compute_instance" "YOUR_RESOURCE_NAME" {

Now we know we need disk_encryption_key_raw field inside boot_disk section.

Basically, you’d have to repeat this process until there is no missing fields remaining.

  • Add missing fields and values
  • run terraform plan
  • Repeat the process until you didn’t see any error.

To make life a little bit easier, here is my Terraform template (resouces.tf) for Google Compute Engine. Please feel free to modify it to suit your need.

// Google Compute Engine Instance 
resource "google_compute_instance" "YOUR_RESOURCE_NAME" {
   name = "YOUR_VM_NAME"
   zone = "YOUR_ZONE"
   project = "YOUR_PROJECT_NAME"
   machine_type = "YOUR_MACHINE_TYPE"
   tags = "YOUR_TAGS"
   labels = "YOUR_LABELS"

   metadata = {
     "status-config-url" = "YOUR_GOOGLE_CONFIG_API"
     "status-uptime-deadline" = "420"
     "status-variable-path" = "status"
     "disable-legacy-endpoints" = "YOUR_DISABLE_LEGACY_ENDPOINTS_OPTION"
   }

   # Startup Script
   metadata_startup_script = file("YOUR_PATH_TO_init.tmpl")

   network_interface {
      network = "https://www.googleapis.com/compute/YOUR_API_VERSION/projects/YOUR_PROJECT_NAME/global/networks/default"
      subnetwork = "https://www.googleapis.com/compute/YOUR_API_VERSION/projects/YOUR_PROJECT_NAME/regions/YOUR_REGION/subnetworks/default"
      subnetwork_project = "YOUR_PROJECT_NAME"


      access_config {
        // Ephemeral IP
      }
    }

   boot_disk {
     source = "https://www.googleapis.com/compute/YOUR_API_VERSION/projects/YOUR_PROJECT_NAME/zones/YOUR_ZONE/disks/YOUR_VM_NAME"
     initialize_params {
       image = "YOUR_INSTANCE_IMAGE"
       size = 15
       type = "pd-standard"
     }
   }

   service_account {
     email = "YOUR_ACCOUNT_SERVICE_EMAIL" 
     scopes = [
      "https://www.googleapis.com/auth/cloudruntimeconfig",
      ....,
      YOUR_EMAIL_SCOPES
      "https://www.googleapis.com/auth/monitoring.write",
     ]
   }

   timeouts {}
}

Once you’re doing filling all the missing pieces, re-run terraform plan command.

If you see the following message, congratulations!
This means there is no discrepancy between Terraform state and your Google Cloud infrastructure.

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

resource.google_compute_instance.YOUR_RESOURCE_NAME: Refreshing state... [id=YOUR_VM_NAME]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
I am a professional software/game developer and entrepreneur. I write about technology in weird places, solo traveling, digital nomad-lifestyle, and random ranting topics.

Related Posts

Back To Top