AMMICO/misinformation/display.py
Petr Andriushchenko 2891c8a6ed
add image summary notebook (#57)
* add image summary notebook

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pin deepface version to avoid bug with progress bar after update

* update actions version for checkout and python

* test ci without lavis

* no lavis for ci test

* merging

* return lavis

* change lavis to salesforce-lavis

* change pycocotools install method

* change pycocotools install method

* fix_pycocotools

* Downgrade Python

* back to 3.9 and remove pycocotools dependance

* instrucctions for windows

* missing comma after merge

* lavis only for ubuntu

* use lavis package name in install instead of git

* adding multimodal searching py and notebook

* exclude lavis on windows

* skip import on windows

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reactivate lavis

* Revert "reactivate lavis"

This reverts commit ecdaf9d316e4b08816ba62da5e0482c8ff15b14e.

* Change input format for multimodal search

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix clip models

* account for new interface in init imports

* changed imports bec of lavis/windows

* fix if-else, added clip ViT-L-14=336 model

* fix code smells

* add model change function to summary

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed new model in summary.py

* fixed summary windget

* moved some function to utils

* fixed imort torch in utils

* added test_summary.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed opencv version

* added first test of multimodal_search.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed test

* removed windows in CI and added test in multimodal search

* change lavis from dependencies from pip ro git

* fixed blip2 model in test_multimodal_search.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed test multimodal search on cpu and gpu machines

* added test, fixed dependencies

* add -vv to pytest command in CI

* added test_multimodal_search tests

* fixed tests in test_multimodal_search.py

* fixed tests in test_summary

* changed CI and fixed test_multimodel search

* fixed ci

* fixed error in test multimodal search, changed ci

* added multimodal search test, added windows CI, added picture in test data

* CI debuging

* fixing tests in CI

* fixing test in CI 2

* fixing CI 3

* fixing CI

* added filtering function

* Brought back all tests after CI fixing

* changed CI one pytest by individual tests

* fixed opencv problem

* fix path for text, adjust result for new gcv

* remove opencv

* fixing cv2 error

* added opencv-contrib, change objects_cvlib

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing tests in CI

* fixing CI testing

* cleanup objects

* fixing codecov in CI

* fixing codecov in CI

* run tests together; install opencv last

* update requirements for opencv dependencies

* moved lavis functions from utils to summary

* Remove lavis from utils.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing jupyter

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: iulusoy <inga.ulusoy@uni-heidelberg.de>
2023-03-22 10:28:09 +01:00

76 строки
2.3 KiB
Python

import ipywidgets
from IPython.display import display
import misinformation.faces as faces
import misinformation.text as text
import misinformation.objects as objects
import misinformation.summary as summary
class JSONContainer:
"""Expose a Python dictionary as a JSON document in JupyterLab
rich display rendering.
"""
def __init__(self, data=None):
if data is None:
data = {}
self._data = data
def _repr_json_(self):
return self._data
def explore_analysis(mydict, identify="faces"):
# dictionary mapping the type of analysis to be explored
identify_dict = {
"faces": faces.EmotionDetector,
"text-on-image": text.TextDetector,
"objects": objects.ObjectDetector,
"summary": summary.SummaryDetector,
}
# create a list containing the image ids for the widget
# image_paths = [mydict[key]["filename"] for key in mydict.keys()]
image_ids = [key for key in mydict.keys()]
# Create an image selector widget
image_select = ipywidgets.Select(
options=image_ids, layout=ipywidgets.Layout(width="20%"), rows=20
)
# Set up the facial recognition output widget
output = ipywidgets.Output(layout=ipywidgets.Layout(width="30%"))
# Set up the image selection and display widget
image_widget = ipywidgets.Box(
children=[],
layout=ipywidgets.Layout(width="50%"),
)
# Register the tab switch logic
def switch(_):
# Clear existing output
image_widget.children = ()
output.clear_output()
# Create the new content
image_widget.children = (
ipywidgets.Image.from_file(mydict[image_select.value]["filename"]),
)
# This output widget absorbes print statements that are messing with
# the widget output and cannot be disabled through the API.
with faces.NocatchOutput():
mydict[image_select.value] = identify_dict[identify](
mydict[image_select.value]
).analyse_image()
with output:
display(JSONContainer(mydict[image_select.value]))
# Register the handler and trigger it immediately
image_select.observe(switch, names=("value",), type="change")
switch(None)
# Show the combined widget
return ipywidgets.HBox([image_select, image_widget, output])