Getting Started with Marqo

First, select your platform:

Marqo Cloud

Marqo Open Source

This page contains information about how to get up and running on Marqo Cloud by creating indexes, adding team members and using Marqo Cloud API Keys.

Full code:

If you have any questions or need help, visit our Community and ask in the get-help channel.

Sign up at the Marqo Cloud Console. You can sign up with an email or via Google.

Create API Key

To run Marqo Cloud, you will need a Marqo API Key. To do this,

Sign up, if you haven't already, to Marqo Cloud.
Navigate to API Keys. You can either create a new API key or use the default. Copy this key, you'll need it when building projects with Marqo.

For a full walkthrough on how to find your API key, visit our article.

Simple Search on Marqo Cloud

If you have already visited our Marqo Cloud Quick Start Guide, you can skip this section.

1. Install Marqo

To install marqo, head to your terminal and enter:

pip install marqo

2. Create Marqo Client

Next, we create an instance of a client for interacting with the Marqo API. We'll specify the server URL, which in this case is running on Marqo Cloud at https://api.marqo.ai.

# Create a Marqo client
api_key = "your_api_key"
mq = marqo.Client("https://api.marqo.ai", api_key=api_key)

Note, api_key here is the API Key we obtained above.

This step sets up the client to interact with Marqo Cloud, allowing us to perform various operations such as creating indexes and adding documents.

3. Housekeeping

Before we create a new index, it's good practice to delete any existing index with the same name to avoid conflicts. Here, we are deleting the "movies-index" if it already exists.

# Delete the index if it already exists
try:
    mq.index("movies-index").delete()
except:
    pass

This ensures that we start with a clean slate every time we run our script.

4. Creating a Marqo Index

Next, we create an index named "movies-index" using a specific machine learning model, hf/e5-base-v2. This model is designed to generate embeddings for various types of text inputs. It will be used for vectorizing the documents we add to the index.

# Create an index - Using this model: https://huggingface.co/intfloat/e5-base-v2
mq.create_index("movies-index", model="hf/e5-base-v2")

Creating an index is crucial as it prepares Marqo to store and manage the documents we'll be working with.

Your terminal should start populating with the following:

YYYY-MM-DD HH:MM:SS,mmm logger:'marqo' INFO Current index status: CREATING

This tells us that Marqo is creating our index.

The Marqo Cloud Indexes UI will also populate with this model:

5. Adding Documents to the Marqo Index

Now, we add some movie descriptions to our index. These descriptions will be vectorized and stored in the index, making them searchable. We specify a 'Title' and 'Description' for each movie.

# Add documents (movie descriptions) to the index
mq.index("movies-index").add_documents(
    [
        {
            "Title": "Inception",  # Title of the movie
            "Description": "A mind-bending thriller about dream invasion and manipulation.",  # Movie description
        },
        {
            "Title": "Shrek",
            "Description": "An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.",
        },
        {
            "Title": "Interstellar",
            "Description": "A team of explorers travel through a wormhole in space to ensure humanity's survival.",
        },
        {
            "Title": "The Martian",
            "Description": "An astronaut becomes stranded on Mars and must find a way to survive.",
        },
    ],
    # Specifies which fields of the documents should be used to generate vectors. In this case, 'Description'.
    tensor_fields=["Description"],
)

In this step, we specify that the "Description" field of each document should be used for vector search by including it in the tensor_fields parameter.

If we click on our index, we can see information such as the number of documents added.

As we see in our case, we've added 4 documents.

6. Search!

With our index populated with movie descriptions, we can now perform a search query. Let's search for a movie related to space exploration.

# Perform a search query on the index
results = mq.index("movies-index").search(
    # Our query
    q="Which movie is about space exploration?"
)

This query searches the descriptions in our index for content related to space exploration.

Finally, we print out the search results, including the title, description, and the relevance score for each movie that matches the query.

# Print the search results
for result in results["hits"]:
    print(
        f"Title: {result['Title']}, Description: {result['Description']}. Score: {result['_score']}"
    )

The relevance score (_score) indicates how well each document matches the search query.

Let’s look at the outputs:

Title: Interstellar, Description: A team of explorers travel through a wormhole in space to ensure humanity's survival.. Score: 0.8173517436600624
Title: The Martian, Description: An astronaut becomes stranded on Mars and must find a way to survive.. Score: 0.8081475581626953
Title: Inception, Description: A mind-bending thriller about dream invasion and manipulation.. Score: 0.7978701791216605
Title: Shrek, Description: An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.. Score: 0.7619883916893311

Creating Your Index

You can either create an index as explained above or you can do it manually on the Cloud UI.

To create an index on the Cloud UI, simple click 'Create Index' and you'll be greeted with the following options:

Click 'Create Index' when happy with your configuration.

Sizing Your Index

Marqo allows you to scale your index in the following ways:

Shards: increasing the number of shards allows you to store more vectors.

Replicas: which allow for more requests per second. Having at least 1 replica enables high availability. Replicas do not affect the total number of shards you can store.

Inference nodes: inference nodes run the machine learning models. You'll need to scale the number of inference nodes depending on the size of the machine learning model you're running and the expected RPS.

Name	Description	Approximate number of 768 dim. vectors	Hourly pricing (USD)
marqo.basic	Low cost, recommended for dev/testing workloads	Up to 2,000,000	$0.0593
marqo.balanced	Storage optimised	Up to 16,000,000	$0.8708
marqo.performance	RPS optimised	Up to 16,000,000	$2.1808

For example, if you created an index with 3 balanced storage shards, the index would be able to store up to 3*16,000,000 = 48,000,000 vectors.

Another example, if you created an index with 2 performance storage shards, the index would be able to store up to 2*16,000,000 = 32,000,000 vectors.

Please note that the maximum number of vectors may vary based on the content of other document fields being stored in the index.

Choosing Storage Shard Types

marqo.basic Marqo basic is the cheapest of the shard types. This is good for proof of concept applications and development work. These shards have higher search latency that the other options and each shard has a lower capacity of approximately 2 million 768 dim vectors. These shards cannot have any replicas either so they are not recommended for production applications where high availability is a requirement.

marqo.balanced Marqo balanced is the middle tier of the shard types. This is good for production applications where high availability is a requirement. These shards have lower search latency than marqo.basic and each shard has a higher capacity of approximately 16 million 768 dim vectors. These shards can have replicas so they are suitable for production applications where high availability is a requirement.

marqo.performance Marqo performance is the highest tier of the shard types. This is good for production applications where high availability and the lowest search latency is a requirement, especially for indexes with tens or hundreds of millions of vectors. These shards have the lowest search latency of all the shard types and each shard has a capacity of approximately 16 million 768 dim. vectors. These shards can have replicas so they are suitable for highly available production application with millions of users.For the tutorials in this getting started guide we will only use marqo.basic shards however if you are deploying a production search with many users we recommend using marqo.balanced. For larger enterprises with large number of concurrent searches a marqo.performance shard is likely more suitable.

Choosing Inference Pod Types

Name	Description	Hourly pricing (USD)
marqo.CPU.large	Storage optimised	$0.3187
marqo.GPU	RPS optimised	$0.9717

The inference pod type adjusts the infrastructure that is used for inference. Inference is the process of creating vectors from your data. A more powerful inference node will reduce latency for indexing and search by creating vectors faster. Inference pod type has a particularly big difference on latency for indexing or searching with images. There are two inference pod types available: marqo.CPU.large Marqo CPU Large is the middle tier of the inference pod types. This is suitable for production applications with low latency search. For many applications a large marqo.CPU pod will be sufficient when searching with text however if searching or indexing images and dealing with very high request concurrency these may become too slow. ‍

marqo.GPU Marqo GPU is the highest tier of the inference pod types. This is suitable for production applications with low latency search and high request concurrency. These pods are significantly faster than marqo.CPU pods when indexing or searching with text and/or images.

A common usage pattern is to mix these nodes for different stages of development. For example you can accelerate indexing of images with marqo.GPU pods and then swap to marqo.CPU pods for searching with only text. You can change your inference configuration at any time by editing the index.

Using a Custom Model on Marqo Cloud

1. Upload Your Model to a Cloud Storage

You need to upload your model (*.pt file for open-clip) to a cloud storage (e.g., Amazon S3, GitHub) and use the downloading address to reference it in Marqo.

Note for Hugging Face sentence transformer models, .zip can be used as an extension.

2. Use Your Model on Marqo Cloud

To use your custom model, you need to navigate to Marqo Cloud and click Create Index within the Indexes tab. When you have opened the Create Index dialogue you will then need to open the Show advanced details dropdown. Here, you can define your finetuned model via model and modelProperties. For an example Open CLIP model, the code is:

{
"treatUrlsAndPointersAsImages": true,
"model": "generic-clip-test-model-1",
"modelProperties": {
    "name": "ViT-B-32-quickgelu",
    "dimensions": 512,
    "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_avg-8a00ab3c.pt",
    "type": "open_clip"
},
"normalizeEmbeddings": true
}

Under url enter the URL of the model
Under dimensions enter the number of dimensions. E.g if you have finetuned a ViT-B-32 model the number of dimensions will be 512.

For a sentence transformer model you can use the following:

# load from a public url
{
"model": "your-own-sentence-transformers-model",
"modelProperties": {
    "dimensions": 384,
    "url": "https://path/to/your/sbert/model.zip",
    "type": "hf"
}
}

Note that above the "hf" type is used to denote a Hugging Face sentence transformer model.

If you'd like to use authentication so that you do not expose your model publicly, you can load it in from a store with an authentication key such as an S3 bucket:

{
"model": "your-own-sentence-transformers-model",
"modelProperties": {
    "dimensions": 384,
    "type": "hf",
    "model_location": {
    "s3": {
        "Bucket": "s3_bucket",
        "Key": "s3_object_key"  // a zip file
    },
    "auth_required": true
    }
}
}

3. Create The Index

Click the create index button to begin index creation. Note that if you use a custom model, the radio buttons for "image compatible" and "text optimised" will be greyed out along with the model dropdown. This is the expected behaviour and you can still create the index.

What's the Difference Between api.marqo.ai and the Index Endpoint?

https://api.marqo.ai is Marqo's control plane endpoint. It can be used for creating, listing and deleting indexes created on Marqo Cloud (see the indexes tab for more information). If you use api.marqo.ai with pymarqo, pymarqo will automatically retrieve the index endpoint so that you can search, index data and perform other operations directly. However, if you are not using pymarqo, in order to search, index and perform other operations on your Marqo index, you will need to copy the index endpoint from the console.

Support

Join our Slack community to ask questions and to chat with other community members about ideas!

Code

What's next?

If you're happy with the basics, why don't you try searching with different modalities:

Here you'll find everything you need to get started building your first end-to-end vector search application with our Open Source vector search engine.

This will walk you through setting up Marqo, adding indexes, documents, performing your first search, and other basic operations.

Full code: Marqo Open Source Getting Started Code

If you have any questions or need help, visit our Community and ask in the get-help channel.

Simple Search with Marqo

We first need to install Marqo. If you have followed our Installation Guide, you can skip this step.

Set Up

Marqo requires Docker. To install Docker go to Docker Docs and install for your operating system (Mac, Windows, Linux).
Once Docker is installed, you can use it to run Marqo. First, open the Docker application and then head to your terminal and enter the following:
```
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
```
Once everything is complete, your terminal should look like this:

Start Indexing and Searching!

Let's look at a simple example below:

PythoncURL

Make sure you have you have Python 3 installed. To check enter the following on your terminal:

python3 --version

The output should be similar to this:

Python 3.9.6

Otherwise, visit here to download and install Python 3.

Next, using a new terminal window, use Python's package manager tool, pip, to install the Marqo Python client:

pip install marqo

Once installed, you can start a Python session to interact with marqo. You can do this, either by opening a Python script in an IDE like VSCode or by entering the command python3 in a new terminal. Now we can start interacting with Marqo.

import marqo

# Create a Marqo client
mq = marqo.Client(url="http://localhost:8882")

# Housekeeping - Delete the index if it already exists
try:
    mq.index("my-first-index").delete()
except:
    pass

# Create the index
mq.create_index("my-first-index", model="hf/e5-base-v2")

# Add documents to the index
mq.index("my-first-index").add_documents(
    [
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing Polo's travels",
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection, "
            "mobility, life support, and communications for astronauts",
            "_id": "article_591",
        },
    ],
    tensor_fields=["Description"],
)

# Obtain results for a specific query
results = mq.index("my-first-index").search(
    q="What is the best outfit to wear on the moon?"
)

mq is the client that wraps themarqo API
create_index() creates a new index with default settings. We optionally specify the model to be hf/e5-base-v2 which is also the default model. Other models are supported by Marqo. See here for the full list of models. Experimentation with different models is often required to achieve the best retrieval for your specific use case. Different models also offer a tradeoff between inference speed and relevancy.
add_documents() takes a list of documents, represented as python dicts, for indexing.
The tensor_fields parameter specifies which fields should be indexed as tensor fields, and searchable with vector search.
You can optionally set a document's ID with the special _id field. Otherwise, Marqo will generate one.

cURL -X POST -H 'Content-type: application/json' http://localhost:8882/indexes/my-first-index -d '{
        "model": "hf/e5-base-v2"
    }'

cURL -X POST -H 'Content-type: application/json' http://localhost:8882/indexes/my-first-index/documents -d '{
    "documents":[
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing Polo'\''s travels"
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection, mobility, life support, and communications for astronauts",
            "_id": "article_591"
        }
    ],
    "tensorFields": ["Description"]
}'

cURL -X POST -H 'Content-type: application/json' http://localhost:8882/indexes/my-first-index/search  -d '{
"q":"What is the best outfit to wear on the moon?"
}'

The POST ... http://localhost:8882/indexes/my-first-index request creates a new index with default settings. In the body, we optionally specify the model to be hf/e5-base-v2 which is also the default model. Experimentation with different models is often required to achieve the best retrieval for your specific use case. Different models also offer a tradeoff between inference speed and relevancy. See here for the full list of models.
The POST ... http://localhost:8882/indexes/my-first-index/documents request takes a list of documents, represented as json objects, for indexing
The tensor_fields parameter in the body specifies which fields should be indexed as tensor fields, and searchable with vector search.
You can optionally set a document's ID with the special _id field. Otherwise, Marqo will generate one.

Let's have a look at the results:

Python

# let's print out the results:
import pprint

pprint.pprint(results)

The result:

Output

{
    "hits": [
        {
            "Description": "The EMU is a spacesuit that provides environmental "
                    "protection, mobility, life support, and "
                    "communications for astronauts",
            "Title": "Extravehicular Mobility Unit (EMU)",
            "_highlights": [{
                "Description": "The EMU is a spacesuit that "
                                    "provides environmental protection, "
                                    "mobility, life support, and "
                                    "communications for astronauts"
            }],
            "_id": "article_591",
            "_score": 0.8302064702029864
    },
    {
            "Description": "A 13th-century travelogue describing Polo's travels",
            "Title": "The Travels of Marco Polo",
            "_highlights": [{"Description": "A 13th-century travelogue "
                                            "describing Polo's travels"}],
            "_id": "8ab5ea5a-2c63-42cd-965f-32be58d05a8b",
            "_score": 0.7665057498844796
            }
    ],
    "limit": 10,
    "offset": 0,
    "processingTimeMs": 138,
    "query": "What is the best outfit to wear on the moon?"
}

Each hit corresponds to a document that matched the search query
They are ordered from most to least matching. This can be seen in the _score field
limit is the maximum number of hits to be returned. This can be set as a parameter during search
Each hit has a _highlights field. This was the part of the document that matched the query the best

Other Basic Operations

Get Document

Retrieve a document by ID.

Python

result = mq.index("my-first-index").get_document(document_id="article_591")

Note that by adding the document using add_documents again using the same _id will cause a document to be updated.

Get Index Stats

Get information about an index.

Python

results = mq.index("my-first-index").get_stats()

Lexical Search

Perform a keyword search. This uses BM25 for the retrieval ranking.

Python

result = mq.index("my-first-index").search("marco polo", search_method="LEXICAL")

Hybrid Search

Perform a hybrid search. This will output a _tensor_score and a _lexical_score.

Python

result = mq.index("my-first-index").search("marco polo", search_method="HYBRID")

To power image and text search, Marqo allows users to plug and play with CLIP models from HuggingFace. Note that if you do not configure multi modal search, image urls will be treated as strings. To start indexing and searching with images, first create an index with a CLIP configuration, as below:

Python

settings = {
    "treat_urls_and_pointers_as_images": True,  # allows us to find an image file and index it
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
}
response = mq.create_index("my-multimodal-index", **settings)

Images can then be added within documents as follows. You can use urls from the internet (for example S3) or from the disk of the machine:

Python

response = mq.index("my-multimodal-index").add_documents(
    [
        {
            "My_Image": "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Hipop%C3%B3tamo_%28Hippopotamus_amphibius%29%2C_parque_nacional_de_Chobe%2C_Botsuana%2C_2018-07-28%2C_DD_82.jpg/640px-Hipop%C3%B3tamo_%28Hippopotamus_amphibius%29%2C_parque_nacional_de_Chobe%2C_Botsuana%2C_2018-07-28%2C_DD_82.jpg",
            "Description": "The hippopotamus, also called the common hippopotamus or river hippopotamus, is a large semiaquatic mammal native to sub-Saharan Africa",
            "_id": "hippo-facts",
        }
    ],
    tensor_fields=["My_Image"],
)

You can then search using text as usual.

Python

results = mq.index("my-multimodal-index").search("animal")

Searching Using an Image

Searching using an image can be achieved by providing the image link.

Python

results = mq.index("my-multimodal-index").search(
    # URL of a Hippo
    "https://upload.wikimedia.org/wikipedia/commons/f/f2/Portrait_Hippopotamus_in_the_water.jpg"
)

Searching Using Weights in Queries

Queries can also be provided as dictionaries where each key is a query and their corresponding values are weights. This allows for more advanced queries consisting of multiple components with weightings towards or against them, queries can have negations via negative weighting.

The example below shows the application of this to a scenario where a user may want to ask a question but also negate results that match a certain semantic criterion.

Python

import marqo
import pprint

mq = marqo.Client(url="http://localhost:8882")

try:
    mq.index("my-weighted-query-index").delete()
except:
    pass

mq.create_index("my-weighted-query-index")

mq.index("my-weighted-query-index").add_documents(
    [
        {
            "Title": "Smartphone",
            "Description": "A smartphone is a portable computer device that combines mobile telephone "
            "functions and computing functions into one unit.",
        },
        {
            "Title": "Telephone",
            "Description": "A telephone is a telecommunications device that permits two or more users to"
            "conduct a conversation when they are too far apart to be easily heard directly.",
        },
        {
            "Title": "Thylacine",
            "Description": "The thylacine, also commonly known as the Tasmanian tiger or Tasmanian wolf, "
            "is an extinct carnivorous marsupial."
            "The last known of its species died in 1936.",
        },
    ],
    tensor_fields=["Description"],
)

# Initially we ask for a type of communications device which is popular in the 21st century
query = {
    # A weighting of 1.1 gives this query slightly more importance
    "I need to buy a communications device, what should I get?": 1.1,
    # This will lead to 'Smartphone' being the top result
    "The device should work like an intelligent computer.": 1.3,
}

results = mq.index("my-weighted-query-index").search(q=query)

print("Query 1:")
pprint.pprint(results)

# Now we ask for a type of communications which predates the 21st century
query = {
    # A weighting of 1 gives this query a neutral importance
    "I need to buy a communications device, what should I get?": 1.0,
    # This will lead to 'Telephone' being the top result
    "The device should work like an intelligent computer.": -0.3,
}

results = mq.index("my-weighted-query-index").search(q=query)

print("\nQuery 2:")
pprint.pprint(results)

Creating and Searching Indexes with Multimodal Combination Fields

Marqo lets you have indexes with multimodal combination fields. Multimodal combination fields can combine text and images into one field. This allows scoring of documents across the combined text and image fields together. It also allows for a single vector representation instead of needing many which saves on storage. The relative weighting of each component can be set per document.

The example below demonstrates this with retrieval of caption and image pairs using multiple types of queries.

Python

import marqo
import pprint

mq = marqo.Client(url="http://localhost:8882")

settings = {
    "treat_urls_and_pointers_as_images": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
}

try:
    mq.index("my-first-multimodal-index").delete()
except:
    pass

mq.create_index("my-first-multimodal-index", **settings)

mq.index("my-first-multimodal-index").add_documents(
    [
        {
            "Title": "Flying Plane",
            "caption": "An image of a passenger plane flying in front of the moon.",
            "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg",
        },
        {
            "Title": "Red Bus",
            "caption": "A red double decker London bus traveling to Aldwych",
            "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image4.jpg",
        },
        {
            "Title": "Horse Jumping",
            "caption": "A person riding a horse over a jump in a competition.",
            "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg",
        },
    ],
    # Note that captioned_image must be a tensor field
    tensor_fields=["captioned_image"],
    # Create the mappings, here we define our captioned_image mapping
    # which weights the image more heavily than the caption - these pairs
    # will be represented by a single vector in the index
    mappings={
        "captioned_image": {
            "type": "multimodal_combination",
            "weights": {"caption": 0.3, "image": 0.7},
        }
    },
)

# Search this index with a simple text query
results = mq.index("my-first-multimodal-index").search(
    q="Give me some images of vehicles and modes of transport. I am especially interested in air travel and commercial aeroplanes."
)

print("Query 1:")
pprint.pprint(results)

# Search the index with a query that uses weighted components
results = mq.index("my-first-multimodal-index").search(
    q={
        "What are some vehicles and modes of transport?": 1.0,
        "Aeroplanes and other things that fly": -1.0,
    }
)
print("\nQuery 2:")
pprint.pprint(results)

Delete documents

Delete documents.

Python

results = mq.index("my-first-index").delete_documents(
    ids=["article_591", "article_602"]
)

Delete index

Delete an index.

Python

results = mq.index("my-first-index").delete()

Support

Join our Slack community to ask questions and to chat with other community members about ideas!

Code

For the code used in this Getting Started Guide, see here.

What's next?

If you're happy with what we've discussed here, try searching with different modalities:

Getting Started with Marqo

Sign up to Marqo Cloud

Create API Key

Simple Search on Marqo Cloud

1. Install Marqo

2. Create Marqo Client

3. Housekeeping

4. Creating a Marqo Index

5. Adding Documents to the Marqo Index

6. Search!

Creating Your Index

Sizing Your Index

Choosing Storage Shard Types

Choosing Inference Pod Types

Using a Custom Model on Marqo Cloud

1. Upload Your Model to a Cloud Storage

2. Use Your Model on Marqo Cloud

3. Create The Index

What's the Difference Between api.marqo.ai and the Index Endpoint?

Support

Code

What's next?

Simple Search with Marqo

Set Up

Start Indexing and Searching!

Other Basic Operations

Get Document

Get Index Stats

Lexical Search

Hybrid Search

Multimodal and Cross Modal Search

Searching Using an Image

Searching Using Weights in Queries

Creating and Searching Indexes with Multimodal Combination Fields

Delete documents

Delete index

Support

Code

What's next?