Using Opentofu to Manage Your Cloud Resources

This page will walk you through using OpenTofu to manage your Marqo Cloud resources.

OpenTofu, a fork of Terraform, is an Open Source infrastructure as code tool that enables you to safely and predictably create, change, and delete cloud resources. For more information on opentofu and terraform, please visit the following links: OpenTofu and Terraform.

The marqo opentofu provider is located at registry.opentofu.org/marqo-ai/marqo.

If you wish to use the terraform provider instead, please note the following - replace the provider source with marqo-ai/marqo - replace all tofu commands in the guide below with terraform

The source code for the OpenTofu/Terraform provider is located at https://github.com/marqo-ai/terraform-provider-marqo.

Installation Instructions

1. Install Opentofu by following the instructions on the Opentofu website. Alternatively, install Terraform by following the instructions on the Terraform website.
2. Create a Marqo configuration (a .tf file with your marqo API and endpoint details) in a new directory
3. Run tofu init in the directory you created the configuration in to initialize the configuration
4. At this point, you should be able to run tofu plan and tofu apply to create and manage your Marqo resources.

Some common commands are: - tofu plan - checks whether the configuration is valid and what actions will be taken - tofu apply - creates or updates the resources in your account - tofu destroy - deletes the resources in your account

See the Opentofu documentation for more information on how to use Opentofu.

Overview of Features

The marqo opentofu provider supports the following:

• A datasource called marqo_read_indices that allows you to read all of your marqo indexes in your account.
• A resource called marqo_index that allows you to create and manage a marqo index.

Sample Configuration

For both of the examples below, create a file within each configuration directory named terraform.tfvars containing your api key as follows

marqo_api_key = "<KEY>"

Note that the host must be set to "https://api.marqo.ai/api/v2"

Reading All Indexes in Your Account (datasource)

terraform {
  required_providers {
    marqo = {
      source = "registry.opentofu.org/marqo-ai/marqo"
      version = "1.0.1"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

data "marqo_read_indices" "example" {
  id = 1
}

output "indices_in_marqo_cloud" {
  value = data.marqo_read_indices.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

Creating and Managing a Structured Index (resource)

terraform {
  required_providers {
    marqo = {
      source = "registry.opentofu.org/marqo-ai/marqo"
      version = "1.0.1"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "example_index_dependent_2"
  settings = {
    type                = "structured"
    vector_numeric_type = "float"
    all_fields = [
      { "name" : "text_field", "type" : "text", "features" : ["lexical_search"] },
      { "name" : "image_field", "type" : "image_pointer" },
      {
        "name" : "multimodal_field",
        "type" : "multimodal_combination",
        "dependent_fields" : {
          "imageField" : 0.8,
          "textField" : 0.1
        },
      },
    ],
    number_of_inferences = 1
    storage_class        = "marqo.basic"
    number_of_replicas   = 0
    number_of_shards     = 2
    tensor_fields        = ["multimodal_field"],
    model                = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings = true
    inference_type       = "marqo.CPU.small"
    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    image_preprocessing = {
      patch_method = null
    }
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

---

Detailed Configuration Options

Required

• api_key (String, Sensitive) The Marqo API key. Can be set with MARQO_API_KEY environment variable.
• host (String) The Marqo API host. Can be set with MARQO_HOST environment variable.

---

marqo_read_indices (Data Source)

Required

• id (String) The unique identifier for the resource.

Read-Only

• items
• last_updated The last time the resource was updated.

Nested Schema for items

• all_fields
• tensor_fields
• ann_parameters
• created The creation date of the index
• docs_count The number of documents in the index
• docs_deleted The number of documents deleted from the index
• filter_string_max_length The filter string max length
• image_preprocessing
• index_name The name of the index
• index_status The status of the index
• inference_type The type of inference used by the index
• marqo_endpoint The Marqo endpoint used by the index
• marqo_version The version of Marqo used by the index
• model The model used by the index
• normalize_embeddings (Boolean) Indicates if embeddings should be normalized
• number_of_inferences The number of inferences made by the index
• number_of_replicas The number of replicas for the index
• number_of_shards The number of shards for the index
• search_query_total The total number of search queries made on the index
• storage_class The storage class of the index
• store_size The size of the index storage
• text_preprocessing (Attributes)
• treat_urls_and_pointers_as_images (Boolean) Indicates if URLs and pointers should be treated as images
• type The type of the index
• vector_numeric_type The numeric type of the vector

Nested Schema for items.all_fields

Read-Only:

• dependent_fields
• features
• name
• type

Nested Schema for items.ann_parameters

Read-Only:

• parameters
• space_type The space type for ANN parameters

Nested Schema for items.ann_parameters.parameters

Read-Only:

• ef_construction The efConstruction parameter for ANN
• m The m parameter for ANN

Nested Schema for items.image_preprocessing

Read-Only:

• patch_method The patch method for image preprocessing

Nested Schema for items.text_preprocessing

Read-Only:

• split_length The split length for text preprocessing
• split_method The split method for text preprocessing
• split_overlap The split overlap for text preprocessing

---

marqo_index (Resource)

For default values in optional fields, please refer to the documentation pages on creating unstructured and structured indexes.

Required

• index_name (String) The name of the index.
• settings (Attributes) The settings for the index.

Nested Schema for settings

Required:

• inference_type (String)
• model (String)
• number_of_inferences (Number)
• number_of_replicas (Number)
• number_of_shards (Number)
• storage_class (String)
• type (String)

Optional:

• all_fields (Attributes List)
• ann_parameters (Attributes)
• filter_string_max_length (Number)
• image_preprocessing (Attributes)
• normalize_embeddings (Boolean)
• tensor_fields (List of String)
• text_preprocessing (Attributes)
• treat_urls_and_pointers_as_images (Boolean)
• vector_numeric_type (String)

Nested Schema for settings.all_fields

Optional:

• dependent_fields (Map of Number)
• features (List of String)
• name (String)
• type (String)

Nested Schema for settings.ann_parameters

Optional:

• parameters (Attributes)
• space_type (String)

Nested Schema for settings.ann_parameters.parameters

Optional:

• ef_construction (Number)
• m (Number)

Nested Schema for settings.image_preprocessing

Optional:

• patch_method (String)

Nested Schema for settings.text_preprocessing

Optional:

• split_length (Number)
• split_method (String)
• split_overlap (Number)