Image summary and visual question answering

This notebooks shows some preliminary work on Image Captioning and Visual question answering with lavis. It is mainly meant to explore its capabilities and to decide on future research directions. We package our code into a misinformation package that is imported here:

[1]:

import misinformation
from misinformation import utils as mutils
from misinformation import display as mdisplay
import misinformation.summary as sm

Set an image path as input file path.

[2]:

images = mutils.find_files(
    path="data/",
    limit=10,
)

[3]:

mydict = mutils.initialize_dict(images)

Create captions for images and directly write to csv

Here you can choose between two models: “base” or “large”

[4]:

summary_model, summary_vis_processors = mutils.load_model("base")
# summary_model, summary_vis_processors = mutils.load_model("large")

[5]:

for key in mydict:
    mydict[key] = sm.SummaryDetector(mydict[key]).analyse_image(
        summary_model, summary_vis_processors
    )

Convert the dictionary of dictionarys into a dictionary with lists:

[6]:

outdict = mutils.append_data_to_dict(mydict)
df = mutils.dump_df(outdict)

Check the dataframe:

[7]:

df.head(10)

[7]:

	filename	const_image_summary	3_non-deterministic summary
0	data/102730_eng.png	two people in blue coats spray disinfection a van	[a couple of people in blue suits and one is s...
1	data/102141_2_eng.png	a collage of images including a corona sign, a...	[various pictures of coronas, including a pers...
2	data/106349S_por.png	a man wearing a face mask while looking at a c...	[a man wearing a face mask is talking on tv, t...

Write the csv file:

[8]:

df.to_csv("./data_out.csv")

Manually inspect the summaries

To check the analysis, you can inspect the analyzed elements here. Loading the results takes a moment, so please be patient. If you are sure of what you are doing.

const_image_summary - the permanent summarys, which does not change from run to run (analyse_image).

3_non-deterministic summary - 3 different summarys examples that change from run to run (analyse_image).

[9]:

mdisplay.explore_analysis(mydict, identify="summary")

[9]:

Generate answers to free-form questions about images written in natural language.

Set the list of questions

[10]:

list_of_questions = [
    "How many persons on the picture?",
    "Are there any politicians in the picture?",
    "Does the picture show something from medicine?",
]

[11]:

for key in mydict:
    mydict[key] = sm.SummaryDetector(mydict[key]).analyse_questions(list_of_questions)

[12]:

mdisplay.explore_analysis(mydict, identify="summary")

[12]:

Convert the dictionary of dictionarys into a dictionary with lists:

[13]:

outdict2 = mutils.append_data_to_dict(mydict)
df2 = mutils.dump_df(outdict2)

[14]:

df2.head(10)

[14]:

	filename	const_image_summary	3_non-deterministic summary	How many persons on the picture?	Are there any politicians in the picture?	Does the picture show something from medicine?
0	data/102730_eng.png	two people in blue coats spray disinfection a van	[a man and woman wearing blue protective gear ...	2	no	yes
1	data/102141_2_eng.png	a collage of images including a corona sign, a...	[various pictures of coronas, including a pers...	1	no	yes
2	data/106349S_por.png	a man wearing a face mask while looking at a c...	[a man wearing a face mask is talking on tv, t...	1	yes	yes

[15]:

df2.to_csv("./data_out2.csv")

[ ]: