AMMICO/notebooks/multimodal_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "22df2297-0629-45aa-b88c-6c61f1544db6",
   "metadata": {},
   "source": [
    "# Image Multimodal Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eeeb302-296e-48dc-86c7-254aa02f2b3a",
   "metadata": {},
   "source": [
    "This notebooks shows some preliminary work on Image Multimodal Search with lavis library. It is mainly meant to explore its capabilities and to decide on future research directions. We package our code into a `misinformation` package that is imported here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f10ad6c9-b1a0-4043-8c5d-ed660d77be37",
   "metadata": {},
   "outputs": [],
   "source": [
    "import misinformation\n",
    "import misinformation.multimodal_search as ms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acf08b44-3ea6-44cd-926d-15c0fd9f39e0",
   "metadata": {},
   "source": [
    "Set an image path as input file path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d3fe589-ff3c-4575-b8f5-650db85596bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "images = misinformation.utils.find_files(\n",
    "    path=\"../data/images/\",\n",
    "    limit=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adf3db21-1f8b-4d44-bbef-ef0acf4623a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "mydict = misinformation.utils.initialize_dict(images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d98b6227-886d-41b8-a377-896dd8ab3c2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "mydict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "987540a8-d800-4c70-a76b-7bfabaf123fa",
   "metadata": {},
   "source": [
    "## Indexing and extracting features from images in selected folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66d6ede4-00bc-4aeb-9a36-e52d7de33fe5",
   "metadata": {},
   "source": [
    "You can choose one of the following models: blip, blip2, albef, clip_base, clip_vitl14, clip_vitl14_336"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bbca1f0-d4b0-43cd-8e05-ee39d37c328e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_type = \"clip_vitl14_336\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca095404-57d0-4f5d-aeb0-38c232252b17",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    model,\n",
    "    vis_processors,\n",
    "    txt_processors,\n",
    "    image_keys,\n",
    "    image_names,\n",
    "    features_image_stacked,\n",
    ") = ms.MultimodalSearch.parsing_images(mydict, model_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ff8a894-566b-4c4f-acca-21c50b5b1f52",
   "metadata": {},
   "source": [
    "The tensors of all images `features_image_stacked` was saved in `<Number_of_images>_<model_name>_saved_features_image.pt`. If you run it once for current model and current set of images you do not need to repeat it again. Instead you can load this features with the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56c6d488-f093-4661-835a-5c73a329c874",
   "metadata": {},
   "outputs": [],
   "source": [
    "# (\n",
    "#    model,\n",
    "#    vis_processors,\n",
    "#    txt_processors,\n",
    "#    image_keys,\n",
    "#    image_names,\n",
    "#    features_image_stacked,\n",
    "# ) = ms.MultimodalSearch.parsing_images(mydict, model_type,\"18_clip_base_saved_features_image.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "309923c1-d6f8-4424-8fca-bde5f3a98b38",
   "metadata": {},
   "source": [
    "Here we already processed our image folder with 18 images with `clip_base` model. So you need just write the name `18_clip_base_saved_features_image.pt` of the saved file that consists of tensors of all images as a 3rd argument to the previous function. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162a52e8-6652-4897-b92e-645cab07aaef",
   "metadata": {},
   "source": [
    "Next, you need to form search queries. You can search either by image or by text. You can search for a single query, or you can search for several queries at once, the computational time should not be much different. The format of the queries is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4196a52-d01e-42e4-8674-5712f7d6f792",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_query3 = [\n",
    "    {\"text_input\": \"politician press conference\"},\n",
    "    {\"text_input\": \"a world map\"},\n",
    "    {\"image\": \"../data/haos.png\"},\n",
    "    {\"image\": \"../data/image-34098-800.png\"},\n",
    "    {\"image\": \"../data/LeonPresserMorocco20032015_600.png\"},\n",
    "    {\"text_input\": \"a dog\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f7dc52f-7ee9-4590-96b7-e0d9d3b82378",
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity = ms.MultimodalSearch.multimodal_search(\n",
    "    mydict,\n",
    "    model,\n",
    "    vis_processors,\n",
    "    txt_processors,\n",
    "    model_type,\n",
    "    image_keys,\n",
    "    features_image_stacked,\n",
    "    search_query3,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1cf7e46-0c2c-4fb2-b89a-ef585ccb9339",
   "metadata": {},
   "source": [
    "After launching `multimodal_search` function, the results of each query will be added to the source dictionary.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad74b21-6187-4a58-9ed8-fd3e80f5a4ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "mydict[\"100127S_ara\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd3ee120-8561-482b-a76a-e8f996783325",
   "metadata": {},
   "source": [
    "A special function was written to present the search results conveniently. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4324e4fd-e9aa-4933-bb12-074d54e0c510",
   "metadata": {},
   "outputs": [],
   "source": [
    "ms.MultimodalSearch.show_results(mydict, search_query3[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86ab96b-1907-4b7f-a78e-3983b516d781",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Save searhing results to csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdbc4d4-695d-4751-ab7c-d2d98e2917d7",
   "metadata": {
    "tags": []
   },
   "source": [
    "Convert the dictionary of dictionarys into a dictionary with lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c6ddd83-bc87-48f2-a8d6-1bd3f4201ff7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "outdict = misinformation.utils.append_data_to_dict(mydict)\n",
    "df = misinformation.utils.dump_df(outdict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2675d5-604c-45e7-86d2-080b1f4559a0",
   "metadata": {
    "tags": []
   },
   "source": [
    "Check the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e78646d6-80be-4d3e-8123-3360957bcaa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05546d99-afab-4565-8f30-f14e1426abcf",
   "metadata": {},
   "source": [
    "Write the csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "185f7dde-20dc-44d8-9ab0-de41f9b5734d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"./data_out.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}