{ "cells": [ { "cell_type": "markdown", "id": "d2c4d40d-8aca-4024-8d19-a65c4efe825d", "metadata": {}, "source": [ "# Facial Expression recognition with DeepFace" ] }, { "attachments": {}, "cell_type": "markdown", "id": "51f8888b-d1a3-4b85-a596-95c0993fa192", "metadata": {}, "source": [ "This notebooks shows some preliminary work on detecting facial expressions with DeepFace. It is mainly meant to explore its capabilities and to decide on future research directions. We package our code into a `ammico` package that is imported here:" ] }, { "cell_type": "code", "execution_count": 1, "id": "b21e52a5-d379-42db-aae6-f2ab9ed9a369", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:32.005061Z", "iopub.status.busy": "2023-05-16T08:49:32.004734Z", "iopub.status.idle": "2023-05-16T08:49:43.843537Z", "shell.execute_reply": "2023-05-16T08:49:43.842770Z" }, "tags": [] }, "outputs": [], "source": [ "import ammico\n", "from ammico import utils as mutils\n", "from ammico import display as mdisplay" ] }, { "cell_type": "markdown", "id": "a2bd2153", "metadata": {}, "source": [ "We select a subset of image files to try facial expression detection on. The `find_files` function finds image files within a given directory:" ] }, { "cell_type": "code", "execution_count": 2, "id": "afe7e638-f09d-47e7-9295-1c374bd64c53", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:43.847941Z", "iopub.status.busy": "2023-05-16T08:49:43.847151Z", "iopub.status.idle": "2023-05-16T08:49:43.851580Z", "shell.execute_reply": "2023-05-16T08:49:43.850932Z" }, "tags": [] }, "outputs": [], "source": [ "images = mutils.find_files(\n", " path=\"data/\",\n", " limit=10,\n", ")" ] }, { "cell_type": "markdown", "id": "e149bfe5-90b0-49b2-af3d-688e41aab019", "metadata": {}, "source": [ "If you want to fine tune the discovery of image files, you can provide more parameters:" ] }, { "cell_type": "code", "execution_count": 3, "id": "f38bb8ed-1004-4e33-8ed6-793cb5869400", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:43.855015Z", "iopub.status.busy": "2023-05-16T08:49:43.854513Z", "iopub.status.idle": "2023-05-16T08:49:43.905661Z", "shell.execute_reply": "2023-05-16T08:49:43.904899Z" } }, "outputs": [], "source": [ "?mutils.find_files" ] }, { "cell_type": "markdown", "id": "705e7328", "metadata": {}, "source": [ "We need to initialize the main dictionary that contains all information for the images and is updated through each subsequent analysis:" ] }, { "cell_type": "code", "execution_count": 4, "id": "b37c0c91", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:43.909380Z", "iopub.status.busy": "2023-05-16T08:49:43.909120Z", "iopub.status.idle": "2023-05-16T08:49:43.912602Z", "shell.execute_reply": "2023-05-16T08:49:43.911886Z" }, "tags": [] }, "outputs": [], "source": [ "mydict = mutils.initialize_dict(images)" ] }, { "cell_type": "markdown", "id": "a9372561", "metadata": {}, "source": [ "To check the analysis, you can inspect the analyzed elements here. Loading the results takes a moment, so please be patient. If you are sure of what you are doing, you can skip this and directly export a csv file in the step below.\n", "Here, we display the face recognition results provided by the DeepFace library. Click on the tabs to see the results in the right sidebar:" ] }, { "cell_type": "code", "execution_count": 5, "id": "992499ed-33f1-4425-ad5d-738cf565d175", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:43.916029Z", "iopub.status.busy": "2023-05-16T08:49:43.915539Z", "iopub.status.idle": "2023-05-16T08:49:45.143021Z", "shell.execute_reply": "2023-05-16T08:49:45.142201Z" }, "tags": [] }, "outputs": [ { "ename": "AttributeError", "evalue": "module 'ammico.display' has no attribute 'explore_analysis'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[5], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mmdisplay\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexplore_analysis\u001b[49m(mydict, identify\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfaces\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", "\u001b[0;31mAttributeError\u001b[0m: module 'ammico.display' has no attribute 'explore_analysis'" ] } ], "source": [ "mdisplay.explore_analysis(mydict, identify=\"faces\")" ] }, { "cell_type": "markdown", "id": "6f974341", "metadata": {}, "source": [ "Directly carry out the analysis and export the result into a csv: Analysis - " ] }, { "cell_type": "code", "execution_count": 6, "id": "6f97c7d0", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:49:45.148322Z", "iopub.status.busy": "2023-05-16T08:49:45.147876Z", "iopub.status.idle": "2023-05-16T08:51:12.701879Z", "shell.execute_reply": "2023-05-16T08:51:12.701032Z" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/serengil/deepface_models/releases/download/v1.0/retinaface.h5' to file '/home/runner/.cache/pooch/3be32af6e4183fa0156bc33bda371147-retinaface.h5'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/chandrikadeb7/Face-Mask-Detection/raw/v1.0.0/mask_detector.model' to file '/home/runner/.cache/pooch/865b4b1e20f798935b70082440d5fb21-mask_detector.model'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/serengil/deepface_models/releases/download/v1.0/age_model_weights.h5' to file '/home/runner/.cache/pooch/39859d3331cd91ac06154cc306e1acc8-age_model_weights.h5'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/serengil/deepface_models/releases/download/v1.0/facial_expression_model_weights.h5' to file '/home/runner/.cache/pooch/dd5d5d6d8f5cecdc0fa6cb34d4d82d16-facial_expression_model_weights.h5'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/serengil/deepface_models/releases/download/v1.0/gender_model_weights.h5' to file '/home/runner/.cache/pooch/2e0d8fb96c5ee966ade0f3f2360f6478-gender_model_weights.h5'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data from 'https://github.com/serengil/deepface_models/releases/download/v1.0/race_model_single_batch.h5' to file '/home/runner/.cache/pooch/382cb5446128012fa5305ddb9d608751-race_model_single_batch.h5'.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 388ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 365ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 368ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 102ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 225ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 251ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 220ms/step\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 212ms/step\n" ] } ], "source": [ "for key in mydict.keys():\n", " mydict[key] = ammico.faces.EmotionDetector(mydict[key]).analyse_image()" ] }, { "cell_type": "markdown", "id": "174357b1", "metadata": {}, "source": [ "Convert the dictionary of dictionarys into a dictionary with lists:" ] }, { "cell_type": "code", "execution_count": 7, "id": "604bd257", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:51:12.720636Z", "iopub.status.busy": "2023-05-16T08:51:12.720112Z", "iopub.status.idle": "2023-05-16T08:51:12.726155Z", "shell.execute_reply": "2023-05-16T08:51:12.725569Z" }, "tags": [] }, "outputs": [], "source": [ "outdict = mutils.append_data_to_dict(mydict)\n", "df = mutils.dump_df(outdict)" ] }, { "cell_type": "markdown", "id": "8373d9f8", "metadata": {}, "source": [ "Check the dataframe:" ] }, { "cell_type": "code", "execution_count": 8, "id": "aa4b518a", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:51:12.729308Z", "iopub.status.busy": "2023-05-16T08:51:12.728803Z", "iopub.status.idle": "2023-05-16T08:51:12.757195Z", "shell.execute_reply": "2023-05-16T08:51:12.756393Z" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filenamefacemultiple_facesno_faceswears_maskagegenderraceemotionemotion (category)
0data/102730_eng.pngYesNo1[No][27][Man][asian][sad][Negative]
1data/102141_2_eng.pngYesNo1[Yes][25][Man][None][None][None]
2data/106349S_por.pngYesNo1[Yes][24][Man][None][None][None]
\n", "
" ], "text/plain": [ " filename face multiple_faces no_faces wears_mask age \n", "0 data/102730_eng.png Yes No 1 [No] [27] \\\n", "1 data/102141_2_eng.png Yes No 1 [Yes] [25] \n", "2 data/106349S_por.png Yes No 1 [Yes] [24] \n", "\n", " gender race emotion emotion (category) \n", "0 [Man] [asian] [sad] [Negative] \n", "1 [Man] [None] [None] [None] \n", "2 [Man] [None] [None] [None] " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(10)" ] }, { "cell_type": "markdown", "id": "579cd59f", "metadata": {}, "source": [ "Write the csv file:" ] }, { "cell_type": "code", "execution_count": 9, "id": "4618decb", "metadata": { "execution": { "iopub.execute_input": "2023-05-16T08:51:12.761552Z", "iopub.status.busy": "2023-05-16T08:51:12.761055Z", "iopub.status.idle": "2023-05-16T08:51:12.768503Z", "shell.execute_reply": "2023-05-16T08:51:12.767818Z" }, "tags": [] }, "outputs": [], "source": [ "df.to_csv(\"data/data_out.csv\")" ] }, { "cell_type": "code", "execution_count": null, "id": "b1a80023", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" }, "vscode": { "interpreter": { "hash": "da98320027a74839c7141b42ef24e2d47d628ba1f51115c13da5d8b45a372ec2" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "45087d2a081b4acfbb2405cb599a0b85": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_965d6ef22b7446618bfb85bff5450ace", "msg_id": "", "outputs": [ { "name": "stdout", "output_type": "stream", "text": "\r1/1 [==============================] - ETA: 0s\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1/1 [==============================] - 1s 750ms/step\n" } ], "tabbable": null, "tooltip": null } }, "514cf50f64854025a217612edd580d0a": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_d656ccd8090d45c98266e3ac6e234984", "msg_id": "", "outputs": [ { "name": "stdout", "output_type": "stream", "text": "\r1/1 [==============================] - ETA: 0s\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1/1 [==============================] - 1s 838ms/step\n" } ], "tabbable": null, "tooltip": null } }, "811abbe79f044ce9bb95c4932afc657e": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "8254cd1952f049e38d288342c609c705": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_811abbe79f044ce9bb95c4932afc657e", "msg_id": "", "outputs": [ { "name": "stdout", "output_type": "stream", "text": "\r1/1 [==============================] - ETA: 0s\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1/1 [==============================] - 1s 786ms/step\n" } ], "tabbable": null, "tooltip": null } }, "965d6ef22b7446618bfb85bff5450ace": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "d656ccd8090d45c98266e3ac6e234984": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }