terraform Archives - ProdSens.live https://prodsens.live/tag/terraform/ News for Project Managers - PMI Thu, 27 Jun 2024 02:20:44 +0000 en-US hourly 1 https://wordpress.org/?v=6.5.5 https://prodsens.live/wp-content/uploads/2022/09/prod.png terraform Archives - ProdSens.live https://prodsens.live/tag/terraform/ 32 32 Deploy Terraform resources to AWS using GitHub Actions via OIDC https://prodsens.live/2024/06/27/deploy-terraform-resources-to-aws-using-github-actions-via-oidc/?utm_source=rss&utm_medium=rss&utm_campaign=deploy-terraform-resources-to-aws-using-github-actions-via-oidc https://prodsens.live/2024/06/27/deploy-terraform-resources-to-aws-using-github-actions-via-oidc/#respond Thu, 27 Jun 2024 02:20:44 +0000 https://prodsens.live/2024/06/27/deploy-terraform-resources-to-aws-using-github-actions-via-oidc/ deploy-terraform-resources-to-aws-using-github-actions-via-oidc

The article explains how to configure OpenID Connect within your GitHub Actions workflows to authenticate with AWS, so…

The post Deploy Terraform resources to AWS using GitHub Actions via OIDC appeared first on ProdSens.live.

]]>
deploy-terraform-resources-to-aws-using-github-actions-via-oidc

The article explains how to configure OpenID Connect within your GitHub Actions workflows to authenticate with AWS, so that the workflow can access AWS resources. The common use case is define AWS infrastructure as code, using CloudFormation, CDK or Terraform, etc, then sync the infrastructure update in AWS through workflows on each code change commit. As we only focus on the IODC setup here, to make it simple, the demo workflow authenticates to AWS account firstly, then list all S3 buckets in that account.

So, what is OpenID Connect?

OpenID Connect (OIDC) is an identity authentication protocol that is an extension of open authorization (OAuth) 2.0 to standardize the process for authenticating and authorizing users when they sign in to access digital services. OIDC provides authentication, which means verifying that users are who they say they are.

Let’s talk about how OpenID Connect works with identity provider and federation.

In AWS, when managing user identities outside of AWS, you can use identity providers instead of creating IAM users in AWS account. With an identity provider (IdP), you can give these external user identities permissions (defined in IAM role) to use AWS resources in your account. An external IdP provides identity information to AWS using either OpenID Connect (OIDC) or SAML 2.0. Identity providers help keep your AWS account secure because you don’t have to distribute or embed long-term security credentials, such as access keys, in your application.

In the demo, GitHub is an external identity provider for AWS. GitHub Actions workflows can be treated as external user identities.

The core process is to authenticate with AWS using temporary credentials within your GitHub Actions workflows. It contains the following steps:

  1. Firstly, establish trust between AWS account and GitHub by adding GitHub identity provider in AWS IAM service.
  2. Create IAM role that allows to be assumed by the new added identity provider.
  3. GitHub actions workflow assumes the IAM role:
    • Workflow retrieves a JWT token from GitHub;
    • Workflow makes an AssumeRoleWithWebIdentity call to AWS STS service with JWT token.
    • AWS STS service validates the trust relationship, and returns temporary credentials in AWS that map to the IAM role with permissions to access specific resources in AWS account
  4. Workflow access AWS resources.

Here is a diagram that displays the authentication process.

auth-process

Prerequisites

  • An AWS account with permission to create OIDC identity provider, role, attach policy in AWS IAM service.
  • An GitHub account to create a repository and workflows.

Solution Overview

Step 1 & 2 Add GitHub IdP & Create IAM Role in AWS Account

You can follow the process and description from the blog just like what I did. I won’t repeat the process here because the blog is clear and understandable.

After completed, you will have an identity provider named token.actions.githubusercontent.com with OpenID Connect type, and an IAM role named GitHubAction-AssumeRoleWithAction with trust relationship.

Identity Provider

IdP

IAM Role -> Trust Relationship

trust-replationship

Step 3. Create GitHub Actions Workflow

In your GitHub repository, create a workflow yaml file named get-started.yaml in .github/workflows directory with below code.

Update and with the real values. Update the branches in the code if you want to trigger the workflow from other branches.

# This is a basic workflow to help you get started with Actions
name: Get Started
# Controls when the action will run. Invokes the workflow on push events but only for the main branch
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
env:
  AWS_REGION: 
  ROLE_TO_ASSUME: arn:aws:iam:::role/GitHubAction-AssumeRoleWithAction
  ROLE_SESSION_NAME: GitHub_to_AWS_via_FederatedOIDC
# Permission can be added at job level or workflow level
permissions:
  id-token: write # This is required for requesting the JWT
  contents: read # This is required for actions/checkout
jobs:
  AssumeRoleAndCallIdentity:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v4
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ env.ROLE_TO_ASSUME }}
          role-session-name: ${{ env.ROLE_SESSION_NAME }}
          aws-region: ${{ env.AWS_REGION }}
      # Hello from AWS: WhoAmI
      - name: Sts GetCallerIdentity
        run: |
          aws sts get-caller-identity

Commit and push to remote. A build is triggered automatically. Below screenshot shows the workflow assumes the IAM role ROLE_TO_ASSUME successfully.

workflow-get-started

At this moment, the IAM role doesn’t attach any policy, which means your workflow has no permissions for AWS resources.

Next, I’m going to use the workflow to list all buckets in my AWS account. To enable it, we need to attach an IAM policy with necessary permission on IAM role GitHubAction-AssumeRoleWithAction.

  1. From AWS IAM Console, find role, Add permissions -> create inline policy.

create-inline-policy

  1. Select a service: S3
  2. Action allowed: ListAllMyBuckets
  3. Resources: All
  4. Next
  5. Policy name: AllowListAllMyBuckets
  6. Create Policy

role-policies

Please be noted, to list all AWS buckets, we choose action s3:ListAllMyBuckets, not s3:ListBucket.

Now, you have attached an inline policy on IAM role. Then add a new step after step Sts GetCallerIdentity to list all buckets using AWS CLI. The step name is List All S3 Buckets, it executes shell script “aws s3 ls” to list all buckets from your AWS account.

...
- name: List All S3 Buckets
  run: aws s3 ls

Commit and push change to remote. A new build is triggered automatically.

workflow-list-all-buckets

Following the least privilege principle, I only granted s3:ListAllMyBuckets permission to the role. The policies assigned to the role determine what the federated users are allowed to do in AWS. You should follow the principle for best practice according to your needs in the daily work.

Others…

Since we are taking about best practice, let’s enhance our workflow by replacing hard-coded or sensitive environment variables with GitHub Secrets and variables. I’m going to save these environment variables in Settings -> Secrets and variables -> Actions. Add repository variables.

add-variables

Finally replace the hard-coded environment variables in workflow.

Hard-coded:

AWS_REGION: xxxxxx
ROLE_TO_ASSUME: arn:aws:iam::xxxxxxxxxxx:role/GitHubAction-AssumeRoleWithAction
ROLE_SESSION_NAME: GitHub_to_AWS_via_FederatedOIDC

Retrieved from GitHub repository variables

AWS_REGION: ${{ vars.AWS_REGION }}
ROLE_TO_ASSUME: ${{ vars.ROLE_TO_ASSUME }}
ROLE_SESSION_NAME: ${{ vars.ROLE_SESSION_NAME }}

Commit and push to remote. A new build is triggered automatically. The workflow will retrieve these variables from GitHub Secrets and variables at the beginning of job.

You can find the source code from GitHub repo: https://github.com/camillehe1992/demo-for-aws-deployment-via-oidc

Summary

You have learned how to add GitHub as an identity provider and create an IAM role with correct trust relationship in AWS. Besides, you created a GitHub Actions workflow that authenticates to AWS and list all S3 buckets in the AWS account within the build.

As I mentioned, the common use case is we define AWS infrastructure as code, these AWS resources are provisioned and managed through GitHub repositories and Actions workflows, a.k.a CICD pipelines. All changes are traceable and controllable, easy to repeat and recover with the power of IoC (Infrastructure as Code) and CICD tools.

References

https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html

Thanks for reading!

The post Deploy Terraform resources to AWS using GitHub Actions via OIDC appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/06/27/deploy-terraform-resources-to-aws-using-github-actions-via-oidc/feed/ 0
How To Manage IAM Access Analyzer in AWS Organizations Using Terraform https://prodsens.live/2024/05/27/how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform/?utm_source=rss&utm_medium=rss&utm_campaign=how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform https://prodsens.live/2024/05/27/how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform/#respond Mon, 27 May 2024 16:20:08 +0000 https://prodsens.live/2024/05/27/how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform/ how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform

Introduction Since I began covering the implementation of security controls in AWS, I have provided walkthroughs on configuring…

The post How To Manage IAM Access Analyzer in AWS Organizations Using Terraform appeared first on ProdSens.live.

]]>
how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform

Introduction

Since I began covering the implementation of security controls in AWS, I have provided walkthroughs on configuring Amazon GuardDuty and AWS Security Hub in a centralized setup using Terraform. In this blog post, we will explore another security service: AWS IAM Access Analyzer. This service helps identify unintended external access or unused access within your organization. Setting up IAM Access Analyzer is simpler than the other services, so let’s dive right in!

About the use case

AWS Identity and Access Management (IAM) Access Analyzer is a feature of AWS IAM that identifies resources shared with external entities and detects unused access, enabling you to mitigate any unintended or obsolete permissions.

IAM Access Analyzer can be used in AWS Organizations, allowing analyzers that use the organization as the zone of trust to be managed by either the management account or a delegated administrator account. This enables the consolidation of findings, which can then be ingested by AWS Security Hub in a centralized setup.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure IAM Access Analyzer in Terraform:

Control Tower standard OU and account structure

The relevant accounts for our use case in the landing zone are:

  1. The Management account for the organization where AWS Organizations is configured. For details, refer to Settings for IAM Access Analyzer.

  2. The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate IAM Access Analyzer administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. With that said, let’s see how we can achieve this using Terraform!

Designating an IAM Access Analyzer administrator account

The IAM Access Analyzer delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To simplify the setup, we will use a multi-provider approach by defining two providers: one for the Management account and another for the Audit account. We will use AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

Unlike other security services that have specific Terraform resources for designating a delegated administrator, this is done using the more general aws_organizations_delegated_administrator resource as follows:

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_organizations_delegated_administrator" "this" {
  provider          = aws.management
  account_id        = data.aws_caller_identity.audit.account_id
  service_principal = "access-analyzer.amazonaws.com"
}

With the Audit account designated as the IAM Access Analyzer administrator, we can now create the analyzers for the organization.

Creating analyzers with organizational zone of trust

As mentioned earlier, there are two types of analyzers: external access and unused access. To make the setup more configurable, we will add some variables and keep them in a separate file called variables.tf. To create the external access analyzer with the organization as the zone of trust, we can define the Terraform configuration as follows:

# Defined in variables.tf

variable "org_external_access_analyzer_name" {
  description = "The name of the organization external access analyzer."
  type        = string
  default     = "OrgExternalAccessAnalyzer"
}
# Defined in main.tf

resource "aws_accessanalyzer_analyzer" "org_external_access" {
  provider      = aws.audit
  analyzer_name = var.org_external_access_analyzer_name
  type          = "ORGANIZATION"
  depends_on    = [aws_organizations_delegated_administrator.this]
}

Since the unused access analyzer is a paid feature, we ought to make it optional. The Terraform configuration can be defined in the following manner:

# Defined in variables.tf

variable "org_unused_access_analyzer_name" {
  description = "The name of the organization unused access analyzer."
  type        = string
  default     = "OrgUnusedAccessAnalyzer"
}

variable "eanble_unused_access" {
  description = "Whether organizational unused access analysis should be enabled."
  type        = bool
  default     = false
}

variable "unused_access_age" {
  description = "The specified access age in days for which to generate findings for unused access."
  type        = number
  default     = 90
}
resource "aws_accessanalyzer_analyzer" "org_unused_access" {
  provider      = aws.audit
  count         = var.eanble_unused_access ? 1 : 0
  analyzer_name = var.org_unused_access_analyzer_name
  type          = "ORGANIZATION_UNUSED_ACCESS"
  configuration {
    unused_access {
      unused_access_age = var.unused_access_age
    }
  }
  depends_on = [aws_organizations_delegated_administrator.this]
}

✅ You can find the complete Terraform in the GitHub repository that accompanies this blog post.

With the complete Terraform configuration, you can now apply it with the appropriate variable values to establish the Audit account as the delegated administrator and create the analyzers with the organization as the zone of trust.

Additional considerations

IAM Access Analyzer is a regional service, so you must create an analyzer in each region. However, it primarily applies to external access analysis, which examines the policies of regional resources such as S3 buckets and KMS keys. Since unused access analysis works with IAM users and roles, which are global resources, creating multiple unused access analyzers would only increase costs without adding value. Therefore, it is recommended to create one external access analyzer per region and only one unused access analyzer in the home region.

Another consideration is that there are times when the organizational zone of trust is not desirable. For example, if you wish to have full segregation of member accounts because they represent different tenants, then you would actually want analyzers created in each member account with itself as the zone of trust. This unfortunately would have to be managed at a per-account level.

Summary

In this blog post, you learned how to manage IAM Access Analyzer in AWS Organizations using Terraform by defining a delegated administrator and using analyzers with the organization as the zone of trust. If you have also configured AWS Security Hub to operate at the organization level, you can manage IAM Access Analyzer findings across accounts and regions, thereby streamlining your security operations.

I hope you find this blog post helpful. Be sure to keep an eye out for more how-to articles on configuring other AWS security services in Terraform, or learn about other topics like generative AI, on the Avangards Blog.

The post How To Manage IAM Access Analyzer in AWS Organizations Using Terraform appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/05/27/how-to-manage-iam-access-analyzer-in-aws-organizations-using-terraform/feed/ 0
How I Use Terraform to Deploy My AWS Amplify React Project https://prodsens.live/2024/05/21/how-i-use-terraform-to-deploy-my-aws-amplify-react-project/?utm_source=rss&utm_medium=rss&utm_campaign=how-i-use-terraform-to-deploy-my-aws-amplify-react-project https://prodsens.live/2024/05/21/how-i-use-terraform-to-deploy-my-aws-amplify-react-project/#respond Tue, 21 May 2024 04:20:20 +0000 https://prodsens.live/2024/05/21/how-i-use-terraform-to-deploy-my-aws-amplify-react-project/ how-i-use-terraform-to-deploy-my-aws-amplify-react-project

As a Software Engineer, I would like to deploy my AWS Amplify React project using Terraform so that…

The post How I Use Terraform to Deploy My AWS Amplify React Project appeared first on ProdSens.live.

]]>
how-i-use-terraform-to-deploy-my-aws-amplify-react-project

As a Software Engineer, I would like to deploy my AWS Amplify React project using Terraform so that I can automate my workflow and eliminate any manual intervention during deployment.

AWS Amplify is a powerful tool that can help you build and deploy full-stack applications with ease

Terraform is a very powerful tool that allows you to define your infrastructure configurations in code using a declarative configuration language. Terraform enables you to automate the provision and management of your AWS Amplify resources. Once your changes are defined in the declarative configuration language, Terraform will apply the changes automatically reducing any manual or human intervention.

I will be walking you through the process of achieving this task using Terraform. This is an assumption that you already have an understanding of Terraform and AWS.

Step 1

Set up your environment for running Terraform; you can use any CICD tool such as Jenkins, GitHub Actions, Spinnaker, CircleCI, GitLab, Makefile, or any tool that you are mostly comfortable; with running your Terraform project. I will be doing a follow-up article on how to use GitHub Actions for running your Terraform Project.

Step 2

Prerequisite: You need to have AWS CLI running on your workstation. This will enable you to authenticate with AWS.

Step 3

I will be creating a reactjs app for this tutorial using the command below for the article. This is the code that will be deployed on AWS Amplify.

npx create-react-app example_reactjs_amplify

The code can be found here https://github.com/ExitoLab/example_reactjs_amplify

Step 4

The script below creates the AWS Amplify App, the branch to deploy from, and the domain on AWS. You will have to define the values for the variables in a separate file called variables.tf and values.tfvars. Since we have enable_branch_auto_build as true this will automatically deploy the amplify app once it detects that new code has been added to the reactjs project; what this means is that once you push a new line of code to the main branch it will automatically redeploy your Amplify app.

resource "aws_amplify_app" "hello_world_amplify" {
  name       = var.app_name
  repository = var.repository #This will be your reactjs project

  access_token             = var.access_token
  enable_branch_auto_build = true

  # The default build_spec added by the Amplify Console for React.
  build_spec = <<-EOT
    version: 0.1
    frontend:
      phases:
        preBuild:
          commands:
            - yarn install
        build:
          commands:
            - yarn run build
      artifacts:
        baseDirectory: build
        files:
          - '**/*'
      cache:
        paths:
          - node_modules/**/*
  EOT

  # The default rewrites and redirects added by the Amplify Console.
  custom_rule {
    source = "https://dev.to/<*>"
    status = "404"
    target = "https://dev.to/index.html"
  }

  environment_variables = {
    Name           = "hello-world"
    Provisioned_by = "Terraform"
  }
}

resource "aws_amplify_branch" "amplify_branch" {
  app_id            = aws_amplify_app.hello_world_amplify.id
  branch_name       = var.branch_name
  enable_auto_build = true
}

resource "aws_amplify_domain_association" "domain_association" {
  app_id                = aws_amplify_app.hello_world_amplify.id
  domain_name           = var.domain_name
  wait_for_verification = false

  sub_domain {
    branch_name = aws_amplify_branch.amplify_branch.branch_name
    prefix      = var.branch_name
  }

}

Step 5

In the variables.tf file, you will define the values for the variables. The file should include the following code:

variable "token" {
  type        = string
  description = "github token to connect github repo"
  default     = "Your Gitub Token"
}

variable "repository" {
  type        = string
  description = "github repo url"
  default     = "The link to your repo comes here"
}

variable "app_name" {
  type        = string
  description = "AWS Amplify App Name"
  default     = "hello-world"
}

variable "branch_name" {
  type        = string
  description = "AWS Amplify App Repo Branch Name"
  default     = "main"
}

variable "domain_name" {
  type        = string
  default     = "awsamplifyapp.com" #change this to your custom domain
  description = "AWS Amplify Domain Name"
}

Step 6

The output will be defined in the output.tf file. Your output file will be similar to the one below.

output "amplify_app_id" {
  value = aws_amplify_app.hello_world_amplify.id
}

These outputs will be generated from the outputs.tf file and this will provide you with all the necessary information on how to access your AWS Amplify app.

Step 7

You can now run the following command to apply your changes

terraform init
terraform plan -var-file=values.tfvar
terraform apply -var-file=values.tfvar

terraform init this command will initialize your Terraform working directory by downloading any necessary modules/ plugins defined in your configuration files.

terraform plan this command will create an execution plan based on your Terraform configuration.

terraform apply this command executes the changes proposed in the execution plan generated by the terraform plan

Step 8

After the terraform apply command has successfully applied your configuration changes. You can now verify that your Amplify resource has been created successfully by visiting the AWS Management Console.

Image description

Next steps: Once the deployment is completed after running Terraform apply. You can now check the app by looking at the URL with https which is visible in the above screenshot.

Below is a screenshot of the AWS Amplify reactjs app

Image description

In Conclusion, I do hope you find this article useful. It explains the process of deploying React on AWS Amplify using Terraform. Please, follow this link below to check the complete code on GitHub. https://github.com/ExitoLab/example_aws_amplify_terraform.git. My next article will be using GitHub Actions to deploy terraform project on AWS . I will be deploying the terraform code for the AWS Amplify app using GitHub Actions.

The post How I Use Terraform to Deploy My AWS Amplify React Project appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/05/21/how-i-use-terraform-to-deploy-my-aws-amplify-react-project/feed/ 0
Automating Snowflake Resource Deployment using Terraform and GitHub Actions https://prodsens.live/2024/05/18/automating-snowflake-resource-deployment-using-terraform-and-github-actions/?utm_source=rss&utm_medium=rss&utm_campaign=automating-snowflake-resource-deployment-using-terraform-and-github-actions https://prodsens.live/2024/05/18/automating-snowflake-resource-deployment-using-terraform-and-github-actions/#respond Sat, 18 May 2024 13:20:26 +0000 https://prodsens.live/2024/05/18/automating-snowflake-resource-deployment-using-terraform-and-github-actions/ automating-snowflake-resource-deployment-using-terraform-and-github-actions

Lately at work, I have been using Terraform for our Infrastructure as Code (IaC) requirements for AWS workloads.…

The post Automating Snowflake Resource Deployment using Terraform and GitHub Actions appeared first on ProdSens.live.

]]>
automating-snowflake-resource-deployment-using-terraform-and-github-actions

Lately at work, I have been using Terraform for our Infrastructure as Code (IaC) requirements for AWS workloads. As part of this learning journey, I also acquired Terraform Associate certification.

I wanted to explore Terraform for non-AWS use cases. At work, we are building a unified data platform for our data needs using Snowflake. So, I thought I will try to automate Snowflake resource deployments using Terraform.

Snowflake is defined as a cloud native, data platform offered as a SaaS. Lately, in the ever-evolving world of data platforms, Snowflake has emerged as a leading cloud-based data warehousing solution.

Manual provisioning of Snowflake resources like Databases, Schemas, Tables, Grants, Warehouses etc, is time consuming and prone to errors. This is where Infrastructure as Code (IaC) tools like Terraform and CI/CD pipelines using GitHub Actions makes life easier.

Terraform is an open-source, cloud agnostic IaC tool that allows us to define and provision cloud infrastructure using a very high level configuration language. Terraform implements this using plugins called providers.

GitHub Actions enables us to create efficient CI/CD pipelines based on code in GitHub repository.

This blog post demonstrates a step-by-step guide on how to deploy resources to Snowflake using Terraform and GitHub Actions, leveraging our repository cicd-with-terraform-for-snowflake. We will deploy a Database, a Schema, Grants and a Table onto two different environments (DEV and PROD) on the a Snowflake instance. We will use release based deployment pipelines to deploy to PROD environment.

Some pre-requisites and assumptions:

  • An AWS Account with an S3 bucket and DynamoDB table already provisioned – we will be using these for Terraform remote backend and State locking.
  • AWS credentials (Access Key and Secret Access Key) for the above AWS account configured as GitHub repository secrets.
  • A Snowflake instance, a user with ACCOUNTADMIN permissions and related Key-pair authentication setup. Related private key configured as GitHub repository secret.
  • A Snowflake role TF_READER pre-created in the Snowflake instance. We will be deploying grants for this role using Terraform resources.

Setting up repository:

Clone the repository to your local machine:

git clone https://github.com/shekar-ym/cicd-with-terraform-for-snowflake.git
cd cicd-with-terraform-for-snowflake

Repository Structure:

  • .github/workflows/: Contains the GitHub Actions workflow files that automate the deployment process.
  • dev and prod folders contain the Terraform files for development and production environment respectively.
  • module folder contains the Terraform module definition which will be used for provisioning Snowflake resources like Database, Schema and Tables.

Terrform Modules:

Modules are containers for multiple resources that are used together. Modules are used to package and reuse resource configurations with Terraform.

In our case, we will be using modules to define Snowflake database, schema, grants, table and warehouse resources configuration. This module will be reused to create resource for development and production environments.

For example, below is the module resource configuration for database and schema:

resource "snowflake_database" "tf_database" {
  name                        = var.database
  comment                     = "Database for ${var.env_name}"
  data_retention_time_in_days = var.time_travel_in_days

}

resource "snowflake_schema" "tf_schema" {
  name     = var.schema
  database = snowflake_database.tf_database.name
  comment  = "Schema for ${var.env_name}"
}

Refer to the Github repository for other module resource configurations.

GitHub Actions Workflow:

The workflow will trigger a deployment to DEV environment when you merge any code changes to main branch using a pull request.

The workflow also includes a step for infrastructure code scan to scan Terraform code. This uses Checkov action against infrastructure-as-code, open source packages, container images, and CI/CD configurations to identify misconfigurations, vulnerabilities, and license compliance issues.

    security-scan-terraform-code:
        name: Security scan (terraform code)
        runs-on: ubuntu-latest
        steps:
          - name: Checkout repo
            uses: actions/checkout@v4

          - name: Run Checkov action
            id: checkov
            uses: bridgecrewio/checkov-action@master
            with:
              directory: .
              soft_fail: true
              download_external_modules: true
              framework: terraform

There is a preview step, when you create a pull request – this preview step performs a terraform plan to give you an overview what resources will be deployed or changed.

When you create a release/* branch from main branch, this triggers a deployment to PROD environment.

Deploying the resources to DEV:

Let us make some changes to the Terraform code, push the changes to GitHub repo and create a pull request (PR). Below is how the deployment pipeline looks:

Image description

And below are the steps performed as part of preview:

Image description

Let us merge our pull request to main branch.

Image description

Image description

Here is the output of terraform apply step:

Image description

Let us verify the resources on Snowflake. As you can see, the deployment pipeline created a database(TASTY_BYTES_DEV) and schema(RAW_POS) and a table (MENU)

Image description

A new warehouse was also provisioned.

Image description

Deploying the resources to PROD:

Let us create a release branch from the main branch. This will trigger a deployment to PROD environment.

Image description

As mentioned earlier, there will be an preview step which performs a terraform plan to give you an overview what resources will be deployed or changed.

Since, I have configured environment protection rules, the pipeline stops for a manual approval, before triggering a deploy to PROD.

Image description

Approving this will trigger a deploy to PROD.

Image description

Here is the output of terraform apply step (for PROD):

Image description

Completed pipeline:

Image description

Let us verify the resources on Snowflake for PROD environment. As you can see, the deployment pipeline created a database(TASTY_BYTES_PROD) and schema(RAW_POS) and a table (MENU)

Image description

A new warehouse for PROD was also provisioned.

Image description

Conclusion:

Automating the deployment of Snowflake resources using Terraform and GitHub Actions streamlines the process, reduces the potential for errors, and ensures that infrastructure is managed consistently. This setup not only saves time but also enhances the reliability and reproducibility of deployments. By following the steps outlined in this guide, you can leverage the power of IaC and CI/CD to manage your Snowflake infrastructure efficiently.

Thanks for reading. Please let me know your feedback in comments section.

The post Automating Snowflake Resource Deployment using Terraform and GitHub Actions appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/05/18/automating-snowflake-resource-deployment-using-terraform-and-github-actions/feed/ 0
Simplifying Keycloak Configuration with Terraform and Terragrunt https://prodsens.live/2024/05/05/simplifying-keycloak-configuration-with-terraform-and-terragrunt/?utm_source=rss&utm_medium=rss&utm_campaign=simplifying-keycloak-configuration-with-terraform-and-terragrunt https://prodsens.live/2024/05/05/simplifying-keycloak-configuration-with-terraform-and-terragrunt/#respond Sun, 05 May 2024 00:20:24 +0000 https://prodsens.live/2024/05/05/simplifying-keycloak-configuration-with-terraform-and-terragrunt/ simplifying-keycloak-configuration-with-terraform-and-terragrunt

Keycloak, an open-source identity and access management solution, provides robust authentication and authorization services for modern applications. However,…

The post Simplifying Keycloak Configuration with Terraform and Terragrunt appeared first on ProdSens.live.

]]>
simplifying-keycloak-configuration-with-terraform-and-terragrunt

Keycloak, an open-source identity and access management solution, provides robust authentication and authorization services for modern applications. However, configuring Keycloak instances manually can be tedious and error-prone. In this blog post, we’ll explore how to simplify Keycloak configuration using Terraform and Terragrunt, enabling infrastructure as code (IaC) practices for managing Keycloak realms, clients, users, and more.

Why Terraform and Terragrunt?

Terraform is an open-source tool that lets you manage your infrastructure as code. With Terraform, you can define your infrastructure in simple configuration files, and Terraform will automatically create, manage, and update your infrastructure according to those configurations.

Terragrunt is a thin wrapper for Terraform that provides extra tools for keeping your Terraform configurations DRY, managing remote state, and working with multiple Terraform modules. It helps you maintain clean, DRY, and repeatable Terraform code by providing extra functionality and best practices.

By leveraging both, we can have easy:

  • Infrastructure as Code (IaC): By defining Keycloak configuration as code, we can version control, review changes, and ensure consistency across environments.
  • Modularization: Modularize our Keycloak configuration, making it easier to manage complex setups.
  • State Management: Manage the state of our infrastructure, preventing configuration drift and ensuring that our infrastructure remains in the desired state.

Let us go!

Install Terraform and Terragrunt

Make sure you have Terraform and Terragrunt installed on your machine. You can find installation instructions on the official Terraform and Terragrunt documentation

Project Structure

Creating a one-size-fits-all structure for a Terraform project can be challenging because it largely depends on the specific requirements of each project. Below is the structure I’ve found most suitable for organizing Keycloak components, concepts, and setup.

.
├── README.md                 <- The project overview
├── .tool-versions            <- Used tools versions (managed by asdf. see https://asdf-vm.com) 
├── README.md                 <- The project overview
├── modules                   <- Terraform modules
|   └── common
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── clients
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── my-realm
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
|   └── other
│       ├── provider.tf
│       └── variables.tf
│       └── output.tf
│       └── main.tf
│       └── README.md
│       └── docs/
└── how-to                    <- Documentation
└── stage                     <- Terraform for environment stage
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules
└── prod                      <- Terraform for environment prod
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules

└── local                     <- Terraform for environment local
    ├── .terraform.lock.hcl   <- Terraform lock file
    └── terragrunt.hcl        <- Terragrunt file
    └── env.yaml              <- environment related variables
    └── main.tf               <- environment modules

In this project structure, I've included a modules directory containing a set of modules shared across all environments. Each module includes a main.tf file encapsulating the module's resources, along with input.tf, output.tf, and variable.tf files for easy configuration across different environments.

Common Module

Let's assume that in the common module, we configure realm events by using the jboss-logging event listener with some non-default configurations. Below is an example of how the main.tf file may look:

resource "keycloak_realm_events" "realm_events" {
  realm_id                     = var.realm_id
  events_enabled               = true
  events_expiration            = 1800
  admin_events_enabled         = true
  admin_events_details_enabled = true
  ]

  events_listeners = [
    "jboss-logging"
  ]
}

To include the realm_id variable, it must be defined in the variables.tf file as shown below:

variable "realm_id" {
  description = "Realm ID"
  type        = string
}

And we must configure the used providers in our module. In this case, the provider.tf will look like as:

terraform {
  required_providers {
    keycloak = {
      source = "mrparkers/keycloak"
    }
  }
}

Master Realm Module

In the realm-master module, the main.tf file should reference the common module. Here is an example of how the main.tf file may look:

data "keycloak_realm" "master" {
  realm = "master"
}

module "realm-master" {
  source = "../../modules/common"
  realm_id = data.keycloak_realm.master.id
}

Similarly, the provider.tf file should be configured as follows:

terraform {
  required_providers {
    keycloak = {
      source = "mrparkers/keycloak"
    }
  }
}

Local environment

In the main.tf file, we need to define the master realm in order to reference the realm-master module in our project.

# Define master realm
module "realm-master" {
  source = "../modules/realm-master"
}

Within each environment (e.g., prod, stage and local), there's an env.yaml file containing all the environment-specific variables.

For example, the env.yaml file for the local environment may look like this:

---
environment: local
url: http://localhost:8080/keycloak

And of course, don't forget to include the terragrunt-local.hcl file, which should be defined in the parent module.

include "root" {
  path = find_in_parent_folders("terragrunt-local.hcl")
}

Terragrunt configuration

As mentioned earlier, Terragrunt is highly beneficial for keeping your configuration Don't Repeat Yourself (DRY). For example, the terragrunt.hcl file may look like this:

# Generates the backend for all modules.
remote_state {
  backend = "s3"
  config  = {
    encrypt        = true
    key            = "keycloak/${path_relative_to_include()}/terraform.tfstate"
    region         = ""
    bucket         = "terraform-states"
    dynamodb_table = "terraform-lock"
  }
}

# Read the local "env.yaml" in every environment.
locals {
  vars        = yamldecode(file("${path_relative_to_include()}/env.yaml"))
  environment = local.vars.environment
  url         = local.vars.url
}

# Generate the "provider.tf" file for every module.
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  required_providers {
    keycloak = {
      source  = "mrparkers/keycloak"
      version = "4.4.0"
    }

    http = {
      source  = "hashicorp/http"
      version = "3.2.1"
    }
  }
}

data "http" "config" {
  url = ""
}

provider "keycloak" {
  client_id     = jsondecode(data.http.config.response_body).terraform-client-id
  client_secret = jsondecode(data.http.config.response_body).terraform-client-secret
  url           = "${local.url}"
}

EOF
}

# Generate the "backend.tf" file for every module.
generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  backend "s3" {}
}
EOF
}

If you're familiar with Terragrunt, you'll find the terragrunt.hcl file above quite familiar, except for the http config part, which I'll describe later in section.

In remote_state, we use AWS S3 to store the state of our environment configurations. Additionally, we generate the backend and provider for every environment.

The locals block is mainly used to read the env.yaml file and assign the variables to local variables.

Lastly, the http config is responsible for making an HTTP call to wherever you securely store your environment configurations (at least the Terraform client ID and secret) to pass them to the keycloak provider.

The terragrunt-local.hcl file is similar to the terragrunt.hcl file, except for the remote state configuration, which isn't necessary in this case. The local file is primarily for testing your configuration on a local Keycloak cluster setup.

The Docker Compose

To run our Terraform configurations, we require a Keycloak cluster setup. In below, we use Docker Compose to start a Keycloak cluster, enabling us to run our local Terraform configurations seamlessly against it.

version: '3'

volumes:
  postgres_data:
      driver: local

services:
  postgres:
      image: postgres
      volumes:
        - postgres_data:/var/lib/postgresql/data
      environment:
        POSTGRES_DB: keycloak
        POSTGRES_USER: keycloak
        POSTGRES_PASSWORD: password
      ports:
        - "5432:5432"
  keycloak:
      image: quay.io/keycloak/keycloak:23.0.6
      environment:
        KC_DB_USERNAME: keycloak
        KC_DB_PASSWORD: password
        KC_DB_URL_HOST: postgres
        KC_DB: postgres
        KC_DB_SCHEMA: public
        KC_HTTP_RELATIVE_PATH: /keycloak
        KC_HOSTNAME_ADMIN: 127.0.0.1
        KC_HOSTNAME: localhost
        KEYCLOAK_ADMIN: admin
        KEYCLOAK_ADMIN_PASSWORD: admin
      command:
        - start-dev
      ports:
        - "8080:8080"
        - "8787:8787"
      depends_on:
        - postgres
  config_keycloak:
    image: ubuntu
    volumes:
      - ./keycloak-docker-config.sh:/opt/keycloak-docker-config.sh
    command: ./opt/keycloak-docker-config.sh
    depends_on:
      - keycloak

The keycloak-docker-config.sh script is primarily used to configure a Terraform client with admin privileges, which Terraform will use during its operations.

#!/bin/bash

apt update -y && apt -y install jq curl

until $(curl --output /dev/null --silent --head --fail http://keycloak:8080/keycloak); do
    printf '.'
    sleep 5
done

# Get access token
TOKEN=$( 
  curl -X POST 
    -H "Content-Type: application/x-www-form-urlencoded" 
    -d "client_id=admin-cli" 
    -d "username=admin" 
    -d "password=admin" 
    -d "grant_type=password" 
    "http://keycloak:8080/keycloak/realms/master/protocol/openid-connect/token" | jq -r '.access_token')

# Create Terraform client (terraform/terraform)
curl -X POST 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer ${TOKEN}" 
  -d '{"clientId": "terraform", "name": "terraform", "enabled": true, "publicClient": false, "secret": "terraform", "serviceAccountsEnabled": true}' 
  "http://keycloak:8080/keycloak/admin/realms/master/clients"

# Get the Terraform service account user ID
USER_ID=$( 
  curl -X GET 
    -H "Content-Type: application/json" 
    -H "Authorization: Bearer ${TOKEN}" 
    "http://keycloak:8080/keycloak/admin/realms/master/users?username=service-account-terraform" | jq -r '.[0].id')

# Get the admin role ID
ROLE_ID=$( 
  curl -X GET 
    -H "Content-Type: application/json" 
    -H "Authorization: Bearer ${TOKEN}" 
    "http://keycloak:8080/keycloak/admin/realms/master/roles" | jq -r '.[] | select(.name == "admin") | .id')

# Add the admin role to the Terraform service account user
curl -kv -X POST 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer ${TOKEN}" 
  -d '[{"id":"'"${ROLE_ID}"'", "name":"admin"}]' 
  "http://keycloak:8080/keycloak/admin/realms/master/users/$USER_ID/role-mappings/realm"

To run it, open the terminal and run the below docker compose command

docker-compose up --build

After the docker-compose containers are up and running, navigate to http://localhost:8080/keycloak and log in using admin/admin as credentials. Ensure that the Terraform client is configured within the master realm.

Terraform client

Now it's time to run your Terraform local configurations. Open your terminal and execute the following commands:

# Navigate to the local environment
$ cd local 
# Ensure that Terraform-related files, including the auto-generated backend.tf and provider.tf, are removed
$ rm -r backend.tf provider.tf terraform.tfstate terraform.tfstate.backup .terraform.lock.hcl .terraform 
# Initialize Terraform to create all necessary files
$ terragrunt init --terragrunt-config terragrunt-local.hcl 
# Apply the Terraform configurations
$ terragrunt apply --terragrunt-config terragrunt-local.hcl 

Now, open your web browser and go to http://localhost:8080/keycloak. Log in using your admin credentials and make sure your configurations are properly set up there!

Conclusion

Organizing and managing Keycloak configurations with Terraform can greatly streamline your development process. By following the structure and steps outlined in this guide, you can efficiently set up and maintain your Keycloak environments, ensuring consistency and scalability across your projects. If you have any questions or suggestions, feel free to leave them in the comments below.

I hope you find it useful!

The post Simplifying Keycloak Configuration with Terraform and Terragrunt appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/05/05/simplifying-keycloak-configuration-with-terraform-and-terragrunt/feed/ 0
Updating Azure DevOps Pipelines for Terraform Post SSH-RSA Deprecation https://prodsens.live/2024/04/05/updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation/?utm_source=rss&utm_medium=rss&utm_campaign=updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation https://prodsens.live/2024/04/05/updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation/#respond Fri, 05 Apr 2024 05:20:07 +0000 https://prodsens.live/2024/04/05/updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation/ updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation

Microsoft’s recent announcement regarding the deprecation of SSH-RSA for connecting to Azure Repos in Azure DevOps represents a…

The post Updating Azure DevOps Pipelines for Terraform Post SSH-RSA Deprecation appeared first on ProdSens.live.

]]>
updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation

Microsoft’s recent announcement regarding the deprecation of SSH-RSA for connecting to Azure Repos in Azure DevOps represents a significant shift towards more secure encryption methods. This change, anticipated with the version post-2022.3 of Azure DevOps Server around the end of 2024, aligns with a broader trend of enhancing security protocols across the tech landscape. However, this transition poses challenges, particularly for organizations leveraging a centralized approach for Terraform module management. These modules, typically stored in dedicated repositories and referenced via SSH in other repositories, are essential for provisioning infrastructure for services or applications. The deprecation of SSH-RSA necessitates an adaptation of these practices.

Centralized module strategy

Adopting a centralized module strategy for Terraform comes with significant advantages, including:

  • Single Source of Truth: Centralizing Terraform modules in one repository ensures consistency, minimizes duplication, and simplifies infrastructure code management. Updates or bug fixes to modules can be easily propagated across all dependent services, eliminating the need to update each service’s repository individually.
  • Ease of Updates: Enhancements, bug fixes, or security patches to a module can be seamlessly rolled out across all implementations by simply updating the module reference in the dependent Terraform configurations.

Image description

Terraform Modules

A common practice is to store Terraform module configurations in version control repositories, such as GitHub or Bitbucket. Terraform accesses these modules using Git, cloning the content over HTTPS or SSH. For private repositories, this process necessitates proper authentication:

  • HTTPS: Requires embedding a Personal Access Token (PAT) within the URL, which poses security risks if exposed in publicly accessible or shared codebases.
module "key_vault_key" {
   source = "git::https://@dev.azure.com/GTRekter/Training/_git/Terraform-Modules//keyvault-key?ref=main"
}
  • SSH: Preferred in automated systems due to its non-interactive nature, allowing seamless access to private repositories.
module "key_vault_key" {
    source = "git::git@ssh.dev.azure.com:v3/GTRekter/Training/_git/Terraform-Modules//keyvault-key?ref=main"
}

After executing terraform init, Terraform clones the referenced external repository into the .terraform/modules directory. This local copy is then utilized in the plan and apply stages, ensuring that the Terraform configurations are self-contained and reproducible.

.
├───.terraform
│   ├───modules
│   │   └───key_vault_key
│   │       ├───.pipelines
│   │       ├───aks-cluster
│   │       ├───aks-node-pool
│   │       └───keyvault-key
│   └───providers
│       └───registry.terraform.io
├───pipelines
└───main.tf

SSH and issues with the current Microsoft update

With the deprecation of SSH-RSA, developers have started encountering unexpected access issues with their repositories overnight.

Image description

The problem arises because the agents execute the clone operation over HTTPS, not relying on SSH-RSA keys. The authentication mechanism at play here is OAuth, which is embedded in the pipeline’s configuration as an http.extraheader. However, when SSH is specified as the mechanism for downloading the source code of the Terraform modules, the process fails.

Migrate from SSH-RSA to RSA-SHA2–512

To move to the new ciphers you will first need to generate public/private key pairs for your client. To do so execute the ssh-keygen command with the option -t set to rsa-sha2–512 to specify the type of key to generate. Once completed you will have a *.pub file (public key), and a file without an extension (private key).

$ ssh-keygen -t rsa-sha2-512
Generating public/private rsa-sha2-512 key pair.
Enter file in which to save the key (~/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~.sshid_rsa
Your public key has been saved in ~.sshid_rsa.pub
The key fingerprint is:
SHA256:zWr3tUHtDLjICoxb6+GjacGmHpYgdVcV9SGVEu2htos gtrekter@Sample
The key's randomart image is:
+---[RSA 3072]----+
|        ..oo++o. |
|       .    .o+. |
|  . . .      +.. |
| . . .   o  o... |
|o  .    S o...o .|
|.. .+o   o ..o + |
|  +o..= o +...o o|
| ....=.= oE... o |
| ...+o+..   . .  |
+----[SHA256]-----+

Next, integrate this key with Azure DevOps by setting up a service account associated with the new key, which the agent will use for downloading Terraform modules. Here’s how to set it up:

  • Navigate to Organization Settings in Azure DevOps.

Image description

  • Under Users, select Add User.

Image description

  • Enter the user’s email, assign them to the project, and add them to the Project Contributor group for necessary permissions.

Image description

Note: The new account requires a mailbox since SSH public keys can only be generated via the Azure DevOps UI due to security protocols.

  • Once the account is set up log in with the new user credentials and click the icon next to the avatar and select SSH public keys.

Image description

  • Click + New Key, paste the public key into the Public Key Data field, and click Add.

Image description

With your RSA-SHA2–512 SSH Key ready for Azure authentication, it’s essential to set up another crucial component for SSH configuration: the known_hosts file. This file records the public keys for the servers you connect to via SSH, pairing each server’s public key with its URL and ensure the integrity of the SSH connection by verifying the identity of the servers you’re connecting to, and protecting against Man-in-the-Middle (MitM) attacks. You can generate this file via the following command:

$ ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10

(Optional) Add the Keys to Azure Key Vault

For those utilizing Azure, it’s a smart move to secure sensitive data like SSH keys in the Azure Key Vault. This not only centralizes your secrets management but also bolsters your project’s security posture. Once you created or selected the KeyVault that will store the private SSH key and the known_hosts file, add a new secret for each one of them. For this guide, we’ll name them ado-ssh-key for the SSH private key and ado-ssh-host-key for the known_hosts file.

Image description

Note: Since the SSH key is multiline, ensure you replace new lines with n to prevent formatting issues that could corrupt the key.

To enable your Azure DevOps agent to access these secrets at runtime, a service connection to Azure is necessary. To do so:

  • In your Azure DevOps project, navigate to Project settings, select Service connections and then click New Service connection.

Image description

Choose Azure Resource Manager and follow the prompts to select your subscription.

Image description

Behind the scenes Azure DevOps will automatically create and configure a service principal in Azure.

Image description

With the Key Vault and service connection ready, the final step involves updating your pipeline to fetch the SSH keys before initiating tasks like terraform init. Insert the following task at the appropriate point in your pipeline:

- task: AzureKeyVault@1
  displayName: 'Get Azure DevOps SSH key and Host Key from KeyVault'
  inputs:
    azureSubscription: VSES Training
    KeyVaultName: kv-training-ssh-dev-krc
    SecretsFilter: 'ado-ssh-key, ado-ssh-host-key'

Setting up the agent SSH key

With the new SSH keys generated using RSA-SHA2–512 SSH, the public key linked to an Azure DevOps user, it’s time to integrate these components into your pipelines. If you’ve opted not to use Azure Key Vault, Azure DevOps variable groups offer an alternative for storing secret values, including SSH keys and known_hosts files. Remember to reference these in your pipeline to ensure they’re accessible when needed.
Before executing tasks like terraform init, the agent’s SSH environment must be configured with the new keys:

  • Create the SSH Directory and set up a .ssh directory with the correct permissions:
echo "Create SSH directory and set permissions"
mkdir -p ~/.ssh/
chmod 700 ~/.ssh/
  • Generate the private key file, naming it uniquely (e.g., ado_id_rsa) to avoid conflicts with keys from other services:
echo "Set SSH keys"
echo "$(ado-ssh-key)" > ~/.ssh/ado_id_rsa
chmod 600 ~/.ssh/ado_id_rsa
  • Append the known_hosts content coming from Azure KeyVault or Azure DevOps variable group to ensure the agent recognizes trusted hosts:
echo "Set known_hosts entries"
echo "$(ado-ssh-host-key)" >> ~/.ssh/known_hosts
  • Create an SSH Config File to instruct SSH to use the specific key (in our case called ado_id_rsa) when the target is a specific host.
echo "Create the SSH config file"
echo -e "Host ssh.dev.azure.comn  HostName ssh.dev.azure.comn  User gitn  IdentityFile ~/.ssh/ado_id_rsan  IdentitiesOnly yesnnHost bitbucket.orgn  HostName bitbucket.orgn  User gitn  IdentityFile ~/.ssh/bitbucket_id_rsan  IdentitiesOnly yes" > ~/.ssh/config
chmod 600 ~/.ssh/config
  • Verify that the SSH setup is functioning correctly:
ssh -Tv git@ssh.dev.azure.com || true
echo "SSH connection test completed"

The final task will be the following:

- task: CmdLine@2
  displayName: Setup SSH key
  inputs:
    script: |
      echo "Create SSH directory and set permissions"
      mkdir -p ~/.ssh/
      chmod 700 ~/.ssh/

      echo "Generate and set SSH keys"
      echo "$(ado-ssh-key)" > ~/.ssh/ado_id_rsa
      chmod 600 ~/.ssh/ado_id_rsa

      echo "Update known_hosts entries"
      echo "$(ado-ssh-host-key)" >> ~/.ssh/known_hosts

      echo "Create the SSH config file"
      echo -e "Host ssh.dev.azure.comn  HostName ssh.dev.azure.comn  User gitn  IdentityFile ~/.ssh/ado_id_rsan  IdentitiesOnly yes" > ~/.ssh/config
      chmod 600 ~/.ssh/config

      ssh -Tv git@ssh.dev.azure.com || true
      echo "SSH connection test completed"

With these configurations in place, your Azure DevOps agent is now equipped to securely connect and deploy your Terraform infrastructure using the updated SSH keys.

Image description

Reference

The post Updating Azure DevOps Pipelines for Terraform Post SSH-RSA Deprecation appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/04/05/updating-azure-devops-pipelines-for-terraform-post-ssh-rsa-deprecation/feed/ 0
Terraform & HashiCorp Vault Integration: Seamless Secrets Management https://prodsens.live/2024/03/23/terraform-hashicorp-vault-integration-seamless-secrets-management/?utm_source=rss&utm_medium=rss&utm_campaign=terraform-hashicorp-vault-integration-seamless-secrets-management https://prodsens.live/2024/03/23/terraform-hashicorp-vault-integration-seamless-secrets-management/#respond Sat, 23 Mar 2024 05:20:48 +0000 https://prodsens.live/2024/03/23/terraform-hashicorp-vault-integration-seamless-secrets-management/ terraform-&-hashicorp-vault-integration:-seamless-secrets-management

what is Hashicorp vault.? Imagine HashiCorp Vault as a secure digital vault for all your sensitive information like…

The post Terraform & HashiCorp Vault Integration: Seamless Secrets Management appeared first on ProdSens.live.

]]>
terraform-&-hashicorp-vault-integration:-seamless-secrets-management

what is Hashicorp vault.?

Imagine HashiCorp Vault as a secure digital vault for all your sensitive information like passwords, API keys, and encryption certificates. It acts like a central location to store, access, and manage these secrets. Here’s why it’s useful:

  • Strong Security: Vault encrypts your secrets and controls access with different permission levels. This minimizes the risk of unauthorized access and keeps your sensitive information safe.

  • Centralized Management: No more scattered secrets! Vault keeps everything in one place, making it easier to control and audit who can access what.

  • It can store various secrets, from database credentials to cloud API keys. Vault also integrates with different tools and platforms you might already be using.

Overall, using Vault helps organizations improve security, simplify secret management, and streamline access control for critical information. While Vault has a free open-source version with limited features, most businesses opt for the paid edition with additional functionalities like advanced auditing and disaster recovery

-1-. In today’s demo, we will explore Terraform Vault integration with a real example and troubleshooting. Let’s get started!

-2-. Launch one EC2 instance with basic configuration and access it.

-3-. Then update your machine and install GPG as well.

sudo apt update && sudo apt install gpg

-4-. Then download the signing key to a new keyring.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

-5-. Then verify the key’s fingerprint.

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

-6-. Clone the Vault repository from GitHub.

mkdir -p $GOPATH/src/github.com/hashicorp && cd $_
git clone https://github.com/hashicorp/vault.git
cd vault

Image description

-7-. Then install Vault.

snap install vault

Image description

-8-. To start Vault, you can use the following command:

vault server -dev -dev-listen-address="0.0.0.0:8200"

Image description

-9-. You may need to set the following environment variables. To do so, create a duplicate session and copy the command, then paste it into the new session.

export VAULT_ADDR='http://0.0.0.0:8200'

Image description

-10- Then go to your EC2 instance’s security group and open port 8200 to access the Vault UI.

Image description

-11-. And then copy the public IP and paste it into your browser’s address bar followed by port 8200. You will see the login page appear.

Image description

-12-. For logging in as root, select ‘Token’ as the default method. Copy the token from your server when you start Vault; it will be displayed to you. After that, you will see the dashboard.

Image description

Image description

-13-. Then, in this demo, we will store the secret as ‘Key & value’. To do so, click on ‘Secret Engines’, then select ‘KV’ which stands for ‘Key/value’, and click on ‘Next’.

Image description

-14-. Next, on the following window, provide a path and click ‘Next’ to proceed, where you will notice that no credentials are created or stored.

Image description

Image description

-15-. Now, click on ‘Create Secret’ and input your secret. I will use ‘Username & password’ as an example.

Image description

Image description

-16-. Then, to access these credentials in Terraform, you need to assign policies or roles similar to IAM. Click on ‘Policies’, then click on ‘Enable New Method’, and select ‘App Role’ as the authentication method. This is similar to IAM. Enable the role. Using this App Role, I will authenticate Terraform or whichever tool you are using.

Image description

Image description

-17-. Then, create a duplicate session of your machine and run the following command on it.

-18-. Because HashiCorp Vault does not support the creation of users in the UI, go to the console and create a user. It’s very easy. Before creating the role, we need to create a policy for that role. This policy enables Terraform to access the ‘kv’ and ‘secret’ folders in Vault.

vault policy write terraform - <

-19-. Now, create a role.

vault write auth/approle/role/terraform 
    secret_id_ttl=10m 
    token_num_uses=10 
    token_ttl=20m 
    token_max_ttl=30m 
    secret_id_num_uses=40 
    token_policies=terraform

Image description

-20-. Similar to AWS, in Vault, we have Role ID and Secret ID. This is sensitive information, so do not share it with anyone.

vault read auth/approle/role/terraform/role-id

vault write -f auth/approle/role/terraform/secret-id

Image description

-21-. Now that we are done with Vault, the next step is to write down the Terraform project and check whether Terraform is able to read the secret from Vault.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = ""
      secret_id = ""
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

-22-. Then, to check if Terraform is able to retrieve the credentials from Vault, run the 'terraform init' and 'terraform apply' commands.

Image description

-23-. Now, let's add the block to create an EC2 instance and apply it again. You will see that in the EC2 tag, our password will be taken.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = ""
      secret_id = ""
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

resource "aws_instance" "example" {
  ami = "ami-0c7217cdde317cfec"
  instance_type = "t2.micro"

  tags = {
    secret = data.vault_kv_secret_v2.example.data["username"]
  }
}

Image description

Image description

Image description

-24-. If you encounter the following error while running 'terraform apply' command, then run it again to recreate 'role-id' and 'secret-id' with the same command as mentioned in step no. 20. After that, paste the new credentials into the code. This will solve your problem.

Image description

The post Terraform & HashiCorp Vault Integration: Seamless Secrets Management appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/03/23/terraform-hashicorp-vault-integration-seamless-secrets-management/feed/ 0
Input Validation – Terraform Tips & Tricks https://prodsens.live/2024/03/10/input-validation-terraform-tips-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=input-validation-terraform-tips-tricks https://prodsens.live/2024/03/10/input-validation-terraform-tips-tricks/#respond Sun, 10 Mar 2024 02:20:32 +0000 https://prodsens.live/2024/03/10/input-validation-terraform-tips-tricks/ input-validation-–-terraform-tips-&-tricks

Terraform modules are a great way to follow D.R.Y. development principles. I’ve written about D.R.Y. in the past,…

The post Input Validation – Terraform Tips & Tricks appeared first on ProdSens.live.

]]>
input-validation-–-terraform-tips-&-tricks

Terraform modules are a great way to follow D.R.Y. development principles. I’ve written about D.R.Y. in the past, but to sum it up briefly:

The DRY (“Don’t Repeat Yourself”) principle follows the idea of every logic duplication being eliminated by abstraction. This means that during the development process we should avoid writing repetitive duplicated code as much as possible.

Writing a module for use by others on your team or the community at large can present some challenges.

If someone isn’t familiar with my code – they may input a value that I did not account for.

Incorrect input can cause deployment failure or, even more frightening, deploy an incorrect configuration without being immediately apparent.

Ralph Wiggum - Danger

In this post, I’ll be covering some of the methods I’ve used to enforce input standards.

Table of Contents

  • Structured Types
  • Validation Blocks

    • Going Further with Functions
    • Adding For Expressions
  • Wrapping Up

Structured Types

You’ll see the most basic form of type enforcement everywhere you look.

The variable below enforces the input standards with the type argument.

This variable will only accept a bool, true or false.

variable "validation_environment" {
  type        = bool
  description = "Set as true to enable validation environment."
  default     = false
}

The input must be a number.

variable "maximum_sessions_allowed" {
  type        = number
  description = "The maximum number of concurrent sessions per host."
  default     = 3
}

You can nest and combine these types using basic Terraform syntax. In the example below, Terraform must receive a list. Even if the list only contains a single string, it is a list of one string.

variable "included_location_ids" {
  type        = list(string)
  description = "A list of Named Location IDs that will this policy will apply against."
  default     = ["All"]
}

In the next example, the type is defined as a map of objects. We’ve added a few other conditions as well – Each objects must have 4 key-value pairs (KPV). Each key must match the defined name, “app_name” or “local_path”. Each value of the key-value pair is also type defined: string, number, bool, etc.

variable "application_map" {
  type = map(object({
    app_name     = string
    local_path   = string
    aad_group    = string
    cmd_argument = string
  }))
  description = "A map of all applications and their metadata."
  default     = null
}

Lets reconsider enforcing the cmd_argument KPV from the object.

variable "application_map" {
  type = map(object({
    app_name     = string
    local_path   = string
    aad_group    = string
  }))
  description = "A map of all applications and their metadata."
  default     = null
}

This change did remove the enforcement of cmd_argument, but it also barred its presence completely.

variable "application_map" {
  type = map(object({
    app_name     = string
    local_path   = string
    aad_group    = string
    cmd_argument = optional(string)
  }))
  description = "A map of all applications and their metadata."
  default     = null
}

The optional modifier allows cmd_argument to be included within the object, but it won’t reject an object without it.

Validation Block

The condition argument succeeds if it evaluates to true. If the statement evaluates to false, Terraform cancels the operation and outputs the string value of the error_message argument.

variable "stars" {
  type        = number
  description = "Please rate your sanctification with my module. Select the number of stars, 1 through 5."
  default     = 5
  validation {
    condition = (
      var.stars >= 1 &&
      var.stars <= 5
    )
    error_message = "Please select a number of stars 1 through 5."
  }
}

Going Further with Functions

We can expand the validation block use with Terraform functions.

In the example below, the anytrue function evaluates the input against each statement. If any of the three statements within evaluate to true, Terraform will proceed.

variable "primary_color" {
  type        = string
  description = "The primary color of your choice!"
  validation {
    condition = anytrue([
      lower(var.primary_color) == "blue",
      lower(var.primary_color) == "red",
      lower(var.primary_color) == "yellow"
    ])
    error_message = "The var.primary_color input is incorrect. Please select blue, red, or yellow."
  }
}

Did you notice the lower function was used here as well? We don’t care if the user inputs “BLUE” or “blue” do we? They’re both valid primary colors.

Adding For Expressions

For Expressions can be used alongside functions to build more thorough validation rules.

  1. Anytrue evaluate every string as it loops each of the ingested list’s values.

  2. The result of each loop is compiled into a new list of bools.

  3. Finally, the list of bools is evaluated against the alltrue function. If any value in the list is false, the condition will fail and trigger error_message.

variable "excluded_platforms" {
  type        = list(string)
  description = "The policy will enforce if the sign-in comes from the listed device platform(s)."
  default     = ["none"]
  validation {
    condition = alltrue([
      for i in var.excluded_platforms : anytrue([
        i == "none",
        i == "all",
        i == "android",
        i == "iOS",
        i == "linux",
        i == "macOS",
        i == "windows",
        i == "windowsPhone",
        i == "unknownFutureValue"
      ])
    ])
    error_message = "Invalid input for included_platforms. The list may only contain the following value(s): none, all, android, iOS, linux, macOS, windows, windowsPhone or unknownFutureValue."
  }
}

Wrapping Up

Validation blocks and structured types are both excellent tools for input validation and consistency.

Mix and match both techniques for the best outcome.

Ralph Wiggum - Goodbye

The post Input Validation – Terraform Tips & Tricks appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/03/10/input-validation-terraform-tips-tricks/feed/ 0
Terraform – Understanding Count and For_Each Loops https://prodsens.live/2024/03/03/terraform-understanding-count-and-for_each-loops/?utm_source=rss&utm_medium=rss&utm_campaign=terraform-understanding-count-and-for_each-loops https://prodsens.live/2024/03/03/terraform-understanding-count-and-for_each-loops/#respond Sun, 03 Mar 2024 18:22:16 +0000 https://prodsens.live/2024/03/03/terraform-understanding-count-and-for_each-loops/ terraform-–-understanding-count-and-for-each-loops

Overview When working with Terraform, you may need to create multiple instances of the same resource. This is…

The post Terraform – Understanding Count and For_Each Loops appeared first on ProdSens.live.

]]>
terraform-–-understanding-count-and-for-each-loops

Overview

When working with Terraform, you may need to create multiple instances of the same resource. This is where count and for_each loops come in. These loops allow you to create multiple resources with the same configuration, but with different values. This guide will explain how to use count and for_each loops in Terraform.

Count in Terraform

The count parameter in Terraform allows you to create a specified number of identical resources. It is an integral part of a resource block that defines how many instances of a particular resource should be created.

Here’s an example of how to use count in Terraform:

resource "azurerm_resource_group" "example" {
  count    = 3
  name     = "resourceGroup-${count.index}"
  location = "East US"
  tags = {
    iteration = "Resource Group number ${count.index}"
  }
}

In the example above, we create three identical resource groups in the Azure region “East US” with differing names using the count parameter.

Pros:

  • Simple to use: The count parameter is straightforward for creating multiple instances of a resource.
  • Suitable for homogeneous resources: When all the resources you’re creating are identical except for an identifier, count is likely a good fit.

Cons:

  • Lacks key-based identification: count doesn’t include a way to address a resource with a unique key directly; you have to rely on an index.
  • Immutable: If you remove an item from the middle of the count list, Terraform marks all subsequent resources for recreation which can be disruptive in certain scenarios. For example: Let’s say you have a Terraform configuration that manages a fleet of virtual machines in Azure using the count parameter. Assume that you initially set the count parameter to 5, which provisioned five VMs:
resource "azurerm_virtual_machine" "vm" {
  count               = 5
  name                = "vm-${count.index}"
  location            = "East US"
  resource_group_name = azurerm_resource_group.rg.name
  network_interface_ids = [azurerm_network_interface.nic[count.index].id]
  # ... (other configuration details)
}

In the above example. Say After some time, you decide that you no longer need the second VM (“vm-1”, since “count.index” is zero-based). To remove this VM, you might change the count to 4 and adjust your resource names or indexes, which might intuitively seem like the correct approach.

The problem arises here: Terraform determines the creation and destruction of resources based on their index. If you simply remove or comment out the definition for “vm-1”, Terraform won’t know that you specifically want to destroy “vm-1”. It would interpret that every VM from index 1 and onward (vm-1, vm-2, vm-3, and vm-4) should be destroyed and recreated because their indices have changed.

This could have several disruptive consequences:

  • Downtime: Recreating VMs would lead to downtime for the services running on them, which may be unacceptable in a production environment.
  • Data Loss: If there’s local data on the VMs that you haven’t backed up, it would be lost when the VMs are destroyed and recreated.
  • IP Changes: If the VMs are assigned dynamic public IPs, these IPs would change and could cause connectivity issues.
  • Costs: Destroying and recreating resources might incur unnecessary costs in terms of the compute hours consumed.

To avoid such issues with count, you’d want to use create_before_destroy lifecycle rules or consider whether for_each is a better choice for such a scenario because it provides a way to uniquely identify resources without relying on sequence. With for_each, each VM would be managed individually, and you could remove a single map entry that corresponds to the unwanted VM, leading to the destruction of only that particular VM without impacting the others.

For_Each in Terraform

The for_each loop in Terraform, used within the for_each argument, iterates over a map or a set of strings, allowing you to create resources that correspond to the given elements.

Here’s an example of how to use for_each in Terraform:

resource "azurerm_resource_group" "example" {
  for_each  = toset(["rg-prod", "rg-dev", "rg-test"])
  name      = each.value
  location  = "East US"
  tags = {
    Name = each.value
  }
}

# Alternatively, using a map
resource "azurerm_storage_account" "example" {
  for_each  = {
    prod = "eastus2"
    dev  = "westus"
    test = "centralus"
  }
  name                     = "storage${each.key}"
  resource_group_name      = azurerm_resource_group.example[each.key].name
  location                 = each.value
  account_tier             = "Standard"
  account_replication_type = "GRS"
}

In the first example using a set of strings, we create resource groups with specific names: “rg-prod”, “rg-dev”, and “rg-test”.

In the second example using a map, we create storage accounts in different locations and with associations to corresponding resource groups.

Pros:

  • Detailed declaration: for_each provides greater control when creating resources that require specific attributes or configurations.
  • Key-based identification: Resources created with for_each can be directly identified and accessed by their keys, making modifications more manageable.
  • Non-destructive updates: If you remove an item from the map or set, only that specific resource will be affected.

Cons:

  • Complexity: for_each is more complex to use than count and requires more planning.
  • Requires a set or map: You must provide a set or map of items to iterate over, which might not be necessary or straightforward for all situations.

When to Use Count vs. For_each

Both constructs are powerful, but they shine in different situations. Here’s a quick reference to determine which to use:

Use Count when:

  • You need to create a fixed number of similar resources.
  • Resource differences can be represented by an index.

Use For_each when:

  • You’re dealing with a collection of items that have unique identifiers.
  • Your resources are not perfectly identical and require individual configurations.
  • You plan to make future modifications that should not affect all resources.

Conclusion

Choosing between count and for_each largely depends on the scenario at hand. The count parameter is excellent for simplicity and when you’re dealing with homogenous resources. Meanwhile, for_each is perfect for a more controlled resource declaration, offering flexibility and precision especially beneficial in complex infrastructures.

Author

Like, share, follow me on: 🐙 GitHub | 🐧 Twitter | 👾 LinkedIn

The post Terraform – Understanding Count and For_Each Loops appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/03/03/terraform-understanding-count-and-for_each-loops/feed/ 0
Writing Python scripts to automate infrastructure in Terraform https://prodsens.live/2024/02/05/writing-python-scripts-to-automate-infrastructure-in-terraform/?utm_source=rss&utm_medium=rss&utm_campaign=writing-python-scripts-to-automate-infrastructure-in-terraform https://prodsens.live/2024/02/05/writing-python-scripts-to-automate-infrastructure-in-terraform/#respond Mon, 05 Feb 2024 11:20:26 +0000 https://prodsens.live/2024/02/05/writing-python-scripts-to-automate-infrastructure-in-terraform/ writing-python-scripts-to-automate-infrastructure-in-terraform

Infrastructure as code (IaC) manages infrastructure through machine-readable definition files rather than physical hardware configuration. This allows infrastructure…

The post Writing Python scripts to automate infrastructure in Terraform appeared first on ProdSens.live.

]]>
writing-python-scripts-to-automate-infrastructure-in-terraform

Infrastructure as code (IaC) manages infrastructure through machine-readable definition files rather than physical hardware configuration. This allows infrastructure to be quickly provisioned, managed, and scaled using code.

Terraform by HashiCorp is a popular open-source infrastructure as a code software tool that provides a consistent CLI and API workflow to manage cloud services, servers, databases, and more from providers like AWS, Azure, and Google Cloud.

Terraform configuration files describe the components needed to run a single application or your entire data center. It handles provisioning and changing resources safely and efficiently.

Deploying cloud-based Python web apps with Streamlit sharing

Check this out: Deploying cloud-based Python web apps with Streamlit sharing

However, regularly running Terraform workflows manually via the CLI can become tedious over time, especially for large infrastructure and frequent updates. This is where automation with Python comes into play.

Python is an accessible programming language commonly used for a wide range of automation tasks. By combining Python scripts with Terraform CLI commands and output values, we can automate our infrastructure workflows for efficiency and consistency.

This allows managing infrastructure-as-code at scale for reduced costs and better reliability.
In this comprehensive guide, we will cover:

  • Prerequisites for using Python with Terraform
  • Executing Terraform functionality like init, plan, and apply from within Python
  • Working with Terraform output values in Python scripts
  • An end-to-end automation walkthrough with Python driving our Terraform config
  • Additional capabilities like testing, modularization, and integration workflows

Prerequisites

To effectively leverage Terraform from Python scripts for infrastructure automation, some key skills are required:

Python Knowledge
You need to be familiar with Python basics like:

  • Variables
  • Control structures like loops and conditionals
  • Functions and modules
  • Importing libraries

Installing Terraform

Terraform CLI should be installed on your system with Python. This allows executing terraform commands from the terminal and scripts.

  • For Linux/MacOS: Have curl and unzip commands available
  • For Windows: Install 7zip for extracting archives
  • Download the package

Next, download the appropriate Terraform package (.zip for Windows, .gz for Linux/Mac) from the Terraform downloads page.

Select a package based on your OS (amd64 for 64-bit operating systems).

  • Extract the Terraform binary

A. For Linux/Mac:

$ tar xvfz terraform_{VERSION}_linux_amd64.zip

This will extract the terraform executable.

B. For Windows, use 7zip or similar.

  • Move Terraform to /usr/local/bin

For easy usage, move Terraform binary file to a location on the OS PATH. Common location is /usr/local/bin/:

$ sudo mv terraform /usr/local/bin/
  • Confirm installation

Check Terraform was installed correctly by checking the version:

$ terraform -v
Terraform v1.1.7

The terraform binary can now be executed directly from the command line and accessed by scripts.

Configuring Terraform Projects

You must have Terraform provider and resource blocks already defined for your infrastructure, such as AWS, Azure, or Google Cloud. Resources like compute instances, databases, and networking must be configured in .tf files for Python to work with.

Infrastructure Concepts
Some infrastructure knowledge is also helpful to ensure you define the right Terraform project architecture and workflows to automate with Python.

Executing Terraform in Python Subprocesses
A common way Python scripts can integrate with external programs and binaries is by using the subprocess module. It allows executing shell commands like terraform and working with the results in Python code.

Here are some key Terraform capabilities we can drive using subprocesses:

  • Initialize Working Directory

The init command reads configuration files and downloads providers needed to provision the infrastructure.

  • Validate Configurations

validate can be used to check for errors in .tf files before applying changes.

  • Plan Infrastructure Changes

To safely preview changes before altering real resources, the Terraform plan shows a diff based on the current state.

  • Apply Changes

The apply command provisions real infrastructure based on your .tf configurations. Resources are created, updated, or destroyed.

  • Destroy Infrastructure

If you no longer require provisioned infrastructure, destroy can dismantle them based on previous applies.

By wrapping these critical Terraform commands in Python subprocess calls, we automate key IaC workflows programmatically. Subprocess attributes also let us parse outputs, check return codes, and handle errors.

  • Working with Terraform Outputs in Python

A key ability enabled by integrating Terraform and Python is to access the output values from Terraform directly applied to Python variables.

Some examples include:

  • Resource identifiers like instance id’s
  • DNS names
  • Security group id’s

We can assign useful output values like:
instance_id = terraform_output['instance_id']

And easily utilize them for subsequent scripts like:

  • Interpolating id’s into other resources
  • Tagging resources in AWS
  • Adding new instances to load balancer pools automatically

Output value integration unlocks many automation use cases by linking Python logic with live infrastructure state data.

Automation Walkthrough

Now let’s walk through a sample automation scenario to see Python and Terraform integration in action:

Our goal is to quickly spin up reusable, best-practice infrastructure. We want to provision an auto-scaled compute cluster in AWS along with networking, load balancing and full monitoring.

We have modular Terraform config files with resources defined as code for each component like EC2, ALB, CloudWatch etc. Our Python script will automatically tie these all together for one command deployment.

The script flow is:

  1. Initialize Terraform working directory
  2. This scans the provider and resource blocks in .tf modules so Terraform understands everything we want to build.
  3. Provision infrastructure
  4. Next we kick off a terraform apply which executes our configurations by communicating to the AWS API and creating real resources like servers, databases, VPCs and so on.
  5. Get resource attributes from state
  6. Once done, we access useful Terraform output values from the newly created infrastructure. For example – VPC id, subnet id’s and all EC2 instance id’s which were dynamically created.
  7. Configure dependencies
  8. Using the retrieved resource attributes and id’s, we can now automatically configure additional dependencies without manual lookup or changing code. For example, we add all EC2 instances to an Auto Scaling group and load balancer for high availability.

To wrap up, destroy the infrastructure to avoid ongoing costs since we were just running a temporary automation test.

By chaining all of these steps driven by the Python script calling Terraform modules, we have an easy way to repeatedly deploy best-practice environments customized to your needs.

Additional Automation Capabilities

There are many more ways Python scrips can provide automation support for Terraform modules beyond the workflow above:

Testing and Validation
Python allows easy integration of unit tests to validate infrastructure changes before deploying. We can also run assertions on output data to ensure resources have been deployed properly.

Reuse and Orchestration
Common modules can be packaged as Python functions for reuse across infrastructure projects. And build orchestration pipelines managing infrastructure code testing > approval > deployment.

State Manipulation
For advanced scenarios, Terraform state can be directly accessed from the Python API for data manipulation or backup.

Conclusion

Python is a versatile way to unlock automation benefits for infrastructure-as-code projects using Terraform for cloud, on-prem, and hybrid environments. Key takeaways:

  • Automate repetitive Terraform workflows like init, plan, and apply
  • Access live state data through output variables in Python
  • Chain together workflows across modules for one-step deployment
  • Reuse Terraform code for consistency and reliability
  • Leverage the Python ecosystem for testing, CI/CD, and reuse

The possibilities for programming infrastructure management are vast to streamline operations.

With robust tooling like Terraform and Python supporting infrastructure-as-code methodologies, teams can codify and automate the provisioning and management of infrastructure at scale for reduced costs and improved efficiency.

If you find this post exciting, find more exciting posts on Learnhub Blog; we write everything tech from Cloud computing to Frontend Dev, Cybersecurity, AI, and Blockchain.

Resource

The post Writing Python scripts to automate infrastructure in Terraform appeared first on ProdSens.live.

]]>
https://prodsens.live/2024/02/05/writing-python-scripts-to-automate-infrastructure-in-terraform/feed/ 0