Image summary and visual question answering

This notebooks shows some preliminary work on Image Captioning and Visual question answering with lavis. It is mainly meant to explore its capabilities and to decide on future research directions. We package our code into a ammico package that is imported here:

[1]:

from ammico import utils as mutils
from ammico import display as mdisplay
import ammico.summary as sm

Set an image path as input file path.

[2]:

images = mutils.find_files(
    path="data/",
    limit=10,
)

[3]:

mydict = mutils.initialize_dict(images)

Create captions for images and directly write to csv

Here you can choose between two models: “base” or “large”

[4]:

obj = sm.SummaryDetector(mydict)
summary_model, summary_vis_processors = obj.load_model("base")
# summary_model, summary_vis_processors = obj.load_model("large")

100%|██████████| 2.50G/2.50G [00:15<00:00, 177MB/s]

[5]:

for key in mydict:
    mydict[key] = sm.SummaryDetector(mydict[key]).analyse_image(
        summary_model, summary_vis_processors
    )

Convert the dictionary of dictionaries into a dictionary with lists:

[6]:

outdict = mutils.append_data_to_dict(mydict)
df = mutils.dump_df(outdict)

Check the dataframe:

[7]:

df.head(10)

[7]:

	filename	const_image_summary	3_non-deterministic summary
0	data/102730_eng.png	two people in blue coats spray disinfection a van	[two people wearing blue uniforms spraying out...
1	data/102141_2_eng.png	a collage of images including a corona sign, a...	[many pictures include people, signs, and a fe...
2	data/106349S_por.png	a man wearing a face mask while looking at a c...	[a man in a face mask talking into the radio, ...

Write the csv file:

[8]:

df.to_csv("./data_out.csv")

Manually inspect the summaries

To check the analysis, you can inspect the analyzed elements here. Loading the results takes a moment, so please be patient. If you are sure of what you are doing.

const_image_summary - the permanent summarys, which does not change from run to run (analyse_image).

3_non-deterministic summary - 3 different summarys examples that change from run to run (analyse_image).

[9]:

mdisplay.explore_analysis(mydict, identify="summary")

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 1
----> 1 mdisplay.explore_analysis(mydict, identify="summary")

AttributeError: module 'ammico.display' has no attribute 'explore_analysis'

Generate answers to free-form questions about images written in natural language.

Set the list of questions

[10]:

list_of_questions = [
    "How many persons on the picture?",
    "Are there any politicians in the picture?",
    "Does the picture show something from medicine?",
]

[11]:

for key in mydict:
    mydict[key] = sm.SummaryDetector(mydict[key]).analyse_questions(list_of_questions)

100%|██████████| 1.35G/1.35G [00:21<00:00, 67.4MB/s]

[12]:

mdisplay.explore_analysis(mydict, identify="summary")

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[12], line 1
----> 1 mdisplay.explore_analysis(mydict, identify="summary")

AttributeError: module 'ammico.display' has no attribute 'explore_analysis'

Convert the dictionary of dictionarys into a dictionary with lists:

[13]:

outdict2 = mutils.append_data_to_dict(mydict)
df2 = mutils.dump_df(outdict2)

[14]:

df2.head(10)

[14]:

	filename	const_image_summary	3_non-deterministic summary	How many persons on the picture?	Are there any politicians in the picture?	Does the picture show something from medicine?
0	data/102730_eng.png	two people in blue coats spray disinfection a van	[two people wearing blue uniforms spraying out...	2	no	yes
1	data/102141_2_eng.png	a collage of images including a corona sign, a...	[many pictures include people, signs, and a fe...	1	no	yes
2	data/106349S_por.png	a man wearing a face mask while looking at a c...	[a man in a face mask talking into the radio, ...	1	yes	yes

[15]:

df2.to_csv("./data_out2.csv")

[ ]: