Skip to content

Bring Your Own Models

Bring your own CLIP model

Most customers can get great performance from publicly available CLIP models. However, some use cases will benefit even more from a model fine-tuned for their domain specific task. In this circumstance, you should use your own model with fine-tuned weights and parameters. It is very convenient to incorporate your own model in Marqo as long as your model belongs to one of the following frameworks:

To use your fine-tuned model, here are the detailed steps:

1. Fine-tune your model

The first step is to fine-tune your model using the frameworks mentioned above. Here we use Open CLIP framework as an example. You should follow the guide to fine-tune your own model and store the trained model (checkpoint) as a *.pt file.

2. Upload your model to a cloud storage

You need to upload your model (the *.pt file) to a cloud storage (e.g., Amazon S3, GitHub) and use the downloading address to reference it in Marqo.

3. Use your model in Marqo

To use your custom model, you need to create an index in Marqo and define it in your index settings via model and modelProperties. For an example Open CLIP model, the code is:

# create an index with your custom model
settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "generic-clip-test-model-1",
    "modelProperties": {
        "name": "ViT-B-32-quickgelu",
        "dimensions": 512,
        "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_avg-8a00ab3c.pt",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip", settings_dict=settings)

You can check Generic CLIP for more detailed information and settings in "modelProperties".

4. Preloading your model

There may be cases wherein you want to preload (or prewarm, in other terms) your model before using it to index. This can be done by adding your model (with model and modelProperties) to the list of models on startup in your marqo configuration.

The syntax for this can be found in Configuring preloaded models

Bring your own Hugging Face Sentence Transformers models

Marqo supports you to use your own Hugging Face Sentence Transformers models. You can use your own model with fine-tuned weights and parameters. It is very convenient to incorporate your own model in Marqo as long as your Sentence Transformers model follows the Hugging Face Sentence Transformers model format.

To use your fine-tuned model, here are the detailed steps:

1. Fine-tune your model

The first step is to fine-tune your model using the sentence-transformers framework. The fine-tuning guide can be found here.

2. Upload your model to a cloud storage

For the Sentence Transformers model, you should include all your files in a directory including the model weights, tokenzier, configurations etc. You should compress the directory into a single file and upload the file to a cloud storage (e.g., Amazon S3, Hugging Face) and use the downloading address to reference it in Marqo. In addition, you can also create a model card in Hugging Face for your fine-tuned and Marqo can use the model card to download your model.

3. Use your model in Marqo

We provide different entries load your own Sentence Transformers model in Marqo, either from a compressed file or from a Hugging Face model card, with or without authentication.

3.1 Load from a compressed file

You can load your own Sentence Transformers model from a compressed file by specifying modelProperties in your index settings.

# load from a public url
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "url": "https://path/to/your/sbert/model.zip",
        "type": "hf",
    },
}
# load from a s3 bucket
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "s3": {
                "Bucket": s3_bucket,
                "Key": s3_object_key,  # a zip file
            },
            "auth_required": True,
        },
    },
}
# load from a Hugging Face zip file
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "hf": {
                "repo_id": hf_repo_name,
                "filename": hf_object,  # a zip file
            },
            "auth_required": True,  # can be True or False
        },
    },
}
response = mq.create_index(
    "your-own-sentence-transformers-model", settings_dict=settings
)

# Loading from a Hugging Face model card with or without authentication using `repo_id` (recommended)
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "hf": {
                "repo_id": hf_repo_name,
            },
            "auth_required": True,  # can be True or False
        },
    },
}
# Loading from a Hugging Face model card without authentication using `name`
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "name": public_repo_name,
        "dimensions": 384,
        "type": "hf",
    },
}
response = mq.create_index(
    "your-own-sentence-transformers-model", settings_dict=settings
)

Please check authentication-in-search and authentication-in-add-documents for the ways to authenticate your model safely in Marqo.

4. Preloading your model

Check here for how to preload your Sentence Transformers models in Marqo.