зеркало из
https://github.com/ssciwr/AMMICO.git
synced 2025-10-29 21:16:06 +02:00
6eff8b8145
* add image summary notebook * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pin deepface version to avoid bug with progress bar after update * update actions version for checkout and python * test ci without lavis * no lavis for ci test * merging * return lavis * change lavis to salesforce-lavis * change pycocotools install method * change pycocotools install method * fix_pycocotools * Downgrade Python * back to 3.9 and remove pycocotools dependance * instrucctions for windows * missing comma after merge * lavis only for ubuntu * use lavis package name in install instead of git * adding multimodal searching py and notebook * exclude lavis on windows * skip import on windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reactivate lavis * Revert "reactivate lavis" This reverts commit ecdaf9d316e4b08816ba62da5e0482c8ff15b14e. * Change input format for multimodal search * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix clip models * account for new interface in init imports * changed imports bec of lavis/windows * fix if-else, added clip ViT-L-14=336 model * fix code smells * add model change function to summary * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed new model in summary.py * fixed summary windget * moved some function to utils * fixed imort torch in utils * added test_summary.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed opencv version * added first test of multimodal_search.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed test * removed windows in CI and added test in multimodal search * change lavis from dependencies from pip ro git * fixed blip2 model in test_multimodal_search.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed test multimodal search on cpu and gpu machines * added test, fixed dependencies * add -vv to pytest command in CI * added test_multimodal_search tests * fixed tests in test_multimodal_search.py * fixed tests in test_summary * changed CI and fixed test_multimodel search * fixed ci * fixed error in test multimodal search, changed ci * added multimodal search test, added windows CI, added picture in test data * CI debuging * fixing tests in CI * fixing test in CI 2 * fixing CI 3 * fixing CI * added filtering function * Brought back all tests after CI fixing * changed CI one pytest by individual tests * fixed opencv problem * fix path for text, adjust result for new gcv * remove opencv * fixing cv2 error * added opencv-contrib, change objects_cvlib * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixing tests in CI * fixing CI testing * fixing codecov in CI * fixing codecov in CI * run tests together; install opencv last * update requirements for opencv dependencies * first doc updates * more changes to doc notebooks --------- Co-authored-by: Petr Andriushchenko <pitandmind@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
73 строки
5.0 KiB
Markdown
73 строки
5.0 KiB
Markdown
# AMMICO - AI Media and Misinformation Content Analysis Tool
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This package extracts data from images such as social media images, and the accompanying text/text that is included in the image. The analysis can extract a very large number of features, depending on the user input.
|
|
|
|
**_This project is currently under development!_**
|
|
|
|
Use pre-processed image files such as social media posts with comments and process to collect information:
|
|
1. Text extraction from the images
|
|
1. Language detection
|
|
1. Translation into English or other languages
|
|
1. Cleaning of the text, spell-check
|
|
1. Sentiment analysis
|
|
1. Subjectivity analysis
|
|
1. Named entity recognition
|
|
1. Topic analysis
|
|
1. Content extraction from the images
|
|
1. Textual summary of the image content ("image caption") that can be analyzed further using the above tools
|
|
1. Feature extraction from the images: User inputs query and images are matched to that query (both text and image query)
|
|
1. Question answering
|
|
1. Performing person and face recognition in images
|
|
1. Face mask detection
|
|
1. Age, gender and race detection
|
|
1. Emotion recognition
|
|
1. Object detection in images
|
|
1. Detection of position and number of objects in the image; currently person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phone
|
|
1. Cropping images to remove comments from posts
|
|
|
|
|
|
## Installation
|
|
|
|
The `AMMICO` package can be installed using pip: Navigate into your package folder `misinformation/` and execute
|
|
```
|
|
pip install .
|
|
```
|
|
This will install the package and its dependencies locally.
|
|
|
|
|
|
## Usage
|
|
|
|
There are sample notebooks in the `misinformation/notebooks` folder for you to explore the package:
|
|
1. Text analysis: Use the notebook `get-text-from-image.ipynb` to extract any text from the images. The text is directly translated into English. If the text should be further analysed, set the keyword `analyse_text` to `True` as demonstrated in the notebook.\
|
|
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/misinformation/blob/main/notebooks/get-text-from-image.ipynb)**
|
|
Place the data files and google cloud vision API key in your google drive to access the data.
|
|
1. Facial analysis: Use the notebook `facial_expressions.ipynb` to identify if there are faces on the image, if they are wearing masks, and if they are not wearing masks also the race, gender and dominant emotion.
|
|
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/misinformation/blob/main/notebooks/facial_expressions.ipynb)**
|
|
Place the data files in your google drive to access the data.**
|
|
1. Object analysis: Use the notebook `ojects_expression.ipynb` to identify certain objects in the image. Currently, the following objects are being identified: person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phone.
|
|
|
|
There are further notebooks that are currently of exploratory nature (`colors_expression.ipynb` to identify certain colors on the image).
|
|
|
|
## Features
|
|
### Text extraction
|
|
The text is extracted from the images using [`google-cloud-vision`](https://cloud.google.com/vision). For this, you need an API key. Set up your google account following the instructions on the google Vision AI website.
|
|
You then need to export the location of the API key as an environment variable:
|
|
`export GOOGLE_APPLICATION_CREDENTIALS="location of your .json"`
|
|
The extracted text is then stored under the `text` key (column when exporting a csv).
|
|
|
|
[Googletrans](https://py-googletrans.readthedocs.io/en/latest/) is used to recognize the language automatically and translate into English. The text language and translated text is then stored under the `text_language` and `text_english` key (column when exporting a csv).
|
|
|
|
If you further want to analyse the text, you have to set the `analyse_text` keyword to `True`. In doing so, the text is then processed using [spacy](https://spacy.io/) (tokenized, part-of-speech, lemma, ...). The English text is cleaned from numbers and unrecognized words (`text_clean`), spelling of the English text is corrected (`text_english_correct`), and further sentiment and subjectivity analysis are carried out (`polarity`, `subjectivity`). The latter two steps are carried out using [TextBlob](https://textblob.readthedocs.io/en/dev/index.html). For more information on the sentiment analysis using TextBlob see [here](https://towardsdatascience.com/my-absolute-go-to-for-sentiment-analysis-textblob-3ac3a11d524).
|
|
|
|
### Emotion recognition
|
|
|
|
### Object detection
|
|
|
|
### Cropping of posts
|