investigation-project-social-unrest.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":5891170,"sourceType":"datasetVersion","datasetId":2501030},{"sourceId":6340043,"sourceType":"datasetVersion","datasetId":3650152},{"sourceId":6413440,"sourceType":"datasetVersion","datasetId":3698851},{"sourceId":6418635,"sourceType":"datasetVersion","datasetId":3702352},{"sourceId":6458626,"sourceType":"datasetVersion","datasetId":3729584},{"sourceId":6495874,"sourceType":"datasetVersion","datasetId":3754476},{"sourceId":6498691,"sourceType":"datasetVersion","datasetId":3756293},{"sourceId":6613514,"sourceType":"datasetVersion","datasetId":3816406}],"dockerImageVersionId":30553,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/liadperetz/protest-signatures?scriptVersionId=149845693\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"**Authors:**\n​\n* [Natan Grayman (2344104)](https://www.kaggle.com/natangrayman)\n* [Liad Peretz (2373287)](https://www.kaggle.com/liadperetz)\n\n\n# **Introduction**\n---\n\nWelcome to our Kaggle notebook dedicated to the investigation of Event Signatures on social media. In this exploration, we delve deep into the world of social media, where we've analyzed Twitter data, including location, news response comparison from GDELT, patterns of interest, and trend validations. This thorough analysis led us to explore pivotal events, and in this notebook, we'll specifically focus on the investigation into the protest event archetype.\n\nFor the protest category, we chose to investigate the following significant events:\n\n**Event 1:**\n\n**The 2021 South African Unrest** — often referred to as the July 2021 riots — unfolded in South Africa's KwaZulu-Natal and Gauteng provinces from **July 9 to 18, 2021**. This wave of civil unrest was triggered by the imprisonment of former President Jacob Zuma for contempt of court.\n\nFor more detailed information about the 2021 South African Unrest, you can visit the [Wikipedia page](https://en.wikipedia.org/wiki/2021_South_African_unrest).\n\n---\n\n**Event 2:** \n\n**The Mahsa Amini Protests in Iran**- began on September 16, 2022, and persisted into 2023, were triggered by the death of Mahsa Amini Iran's morality police for \"improperly\" wearing her hijab. These protests stand out as a unique and powerful expression of dissent, challenging the Iranian government and constituting the most significant revolt the country has witnessed since the 1979 Islamic Revolution.\n\nFor more detailed information about the Mahsa Amini Protests, you can visit the [Wikipedia page](https://en.wikipedia.org/wiki/Mahsa_Amini_protests).\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# **Table of Contents**\n\n[1. Twitter Dataset Analysis](#section-one)\n\n   * [Dataset Summary](#section-one-a)\n   * [Frequency of words in tweets](#section-one-b)\n   * [Number of Tweets per Day](#section-one-c)\n   * [Location Analysis](#section-one-d)\n   \n   \n[2. GDELT Analysis](#section-two)\n   * [GDELT Description](#section-two-a)\n   * [GDELT Exploratory Data Analysis (EDA)](#section-two-b)\n   * [Total Number of Mentions per day](#section-two-c)\n   * [GDELT Sentiment Analysis](#section-two-d)\n   \n   \n   \n[3. Google Trends Data Analysis](#section-three)\n   * [Google Trends Description](#section-three-a)\n   * [Web Search](#section-three-b)\n   * [News Search ](#section-three-c)\n   \n   \n[4. Pattern of Interest Comparisons](#section-four)\n   * [Comparison of Number of Tweets and GDELT Articles per Day](#section-four-a)\n   * [Comparison of Tweets and Google Trends' Web Search](#section-four-b)\n   * [Comparison of GDELT Articles and Google Trends' News Search](#section-four-c)\n    \n  \n[5. Mathematical Framework](#section-five)\n   * [Seasonal Decomposition](#section-five-a)\n   * [LOESS Regression](#section-five-b)\n   * [Cross-Correlation between the News Media and Social Media](#section-five-c)\n    \n\n[6. Iran Protest 2022](#section-six)\n   * [Number of Tweets per Day](#section-six-a)\n   * [Alignment of Signatures](#section-six-b)\n   * [Iran Protests vs SA Unrest](#section-six-c)\n   \n   \n[7. Event Archetype](#section-seven)\n   * [Twitter Signature Similarity Metrics](#section-seven-a)\n   * [Archetype Curves Formation](#section-seven-b)\n   * [Evaluation Metrics](#section-seven-c)\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-11-08T11:16:54.6975Z","iopub.execute_input":"2023-11-08T11:16:54.697903Z","iopub.status.idle":"2023-11-08T11:16:55.152503Z","shell.execute_reply.started":"2023-11-08T11:16:54.697868Z","shell.execute_reply":"2023-11-08T11:16:55.151284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n# **1. Twitter Dataset Analysis**","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-one-a\"></a>\n## Dataset Summary\n\n\n**Source**: The dataset has been collected from an open-source repository available at this site: [Twitter Stream Archive](https://archive.org/search?query=collection%3Atwitterstream&sort=-publicdate).\n\n**Date Range**: The dataset covers a substantial period, ranging from **July 9, 2021**, to **July 21, 2021**.\n\n**Volume**: This dataset is extensive, with approximately **4,000 to 5,000 tweets recorded every minute** during the specified date range.\n\nThe dataset provides a comprehensive collection of tweets, capturing a snapshot of social media conversations during a significant time frame. It offers a valuable resource for analyzing and understanding online discourse during the specified period.\n\n\n\n","metadata":{}},{"cell_type":"code","source":"csv_file_path= '/kaggle/input/all-days-extracted-tweets-unrest-2021/Extracted_south_african_protest_7.csv'\nunrest_tweets2 = pd.read_csv(csv_file_path, low_memory=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:55.154259Z","iopub.execute_input":"2023-11-08T11:16:55.154737Z","iopub.status.idle":"2023-11-08T11:16:55.278165Z","shell.execute_reply.started":"2023-11-08T11:16:55.154704Z","shell.execute_reply":"2023-11-08T11:16:55.277183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unrest_tweets2.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:55.279373Z","iopub.execute_input":"2023-11-08T11:16:55.27989Z","iopub.status.idle":"2023-11-08T11:16:55.318229Z","shell.execute_reply.started":"2023-11-08T11:16:55.279861Z","shell.execute_reply":"2023-11-08T11:16:55.317382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('There are {} rows and {} columns in the twitter dataset'.format(unrest_tweets2.shape[0],unrest_tweets2.shape[1]))","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:55.320521Z","iopub.execute_input":"2023-11-08T11:16:55.321073Z","iopub.status.idle":"2023-11-08T11:16:55.325825Z","shell.execute_reply.started":"2023-11-08T11:16:55.321042Z","shell.execute_reply":"2023-11-08T11:16:55.324875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-one-c\"></a>\n## **Number of Tweets per Day**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Assuming earthquake_tweets is your DataFrame with the 'date' column\nunrest_tweets2['date'] = pd.to_datetime(unrest_tweets2[\"Tweet Created At\"])  # Convert the 'date' column to datetime\n\n# Resample the data by day and count the number of tweets\ntweets_per_day = unrest_tweets2.resample('D', on='date').size()\n\nprint(tweets_per_day)\n\nunrest_tweets_per_day_1 = {\"date\": tweets_per_day.index,\n                          \"count\": tweets_per_day.values}\n\n# Create a figure and axis\nfig, ax = plt.subplots()\n\n# Plot the data\nax.plot(tweets_per_day.index, tweets_per_day.values, marker='o')\n\n# Set the x-axis format to display only the date\ndate_formatter = DateFormatter('%Y-%m-%d')\nax.xaxis.set_major_formatter(date_formatter)\n\n# Set the dates starting from the beginning of the event:\nspecific_dates = [pd.Timestamp('2021-07-09')]  \ndate_range = pd.date_range(start='2021-07-10', end='2021-07-18', freq='D')\nall_dates = specific_dates + date_range.tolist() \n\nax.set_xticks(all_dates)\n\n# # Ensure that the specific date '2023-02-06' is displayed on the x-axis\n# ax.axvline(pd.Timestamp('2023-02-06'), color='red', linestyle='--', label='2023-02-06')\n# ax.legend()\n\n# Rotate the x-axis labels for better visibility\nplt.xticks(rotation=90)\n\n# Set labels and title\nax.set_xlabel('Date')\nax.set_ylabel('Number of Tweets')\nax.set_title('Number of Tweets per Day')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:55.328331Z","iopub.execute_input":"2023-11-08T11:16:55.328653Z","iopub.status.idle":"2023-11-08T11:16:56.736869Z","shell.execute_reply.started":"2023-11-08T11:16:55.328626Z","shell.execute_reply":"2023-11-08T11:16:56.735889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-one-d\"></a>\n## **Location Analysis**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Count non-null values in the \"Tweet Coordinates\" column\nnon_null_count = unrest_tweets2[\"Tweet Coordinates\"].count()\n\n# Display the count\nprint(\"Number of non-null values in 'Tweet Coordinates':\", non_null_count)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.738228Z","iopub.execute_input":"2023-11-08T11:16:56.738581Z","iopub.status.idle":"2023-11-08T11:16:56.744732Z","shell.execute_reply.started":"2023-11-08T11:16:56.738513Z","shell.execute_reply":"2023-11-08T11:16:56.743741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Print the first 5 rows of the \"User Location\" column\nprint(\"Sample of 'User Location' column:\")\nprint(unrest_tweets2[\"User Location\"].head())\n\n# Check the datatype of the \"User Location\" column\ndata_type = unrest_tweets2[\"User Location\"].dtype\nprint(\"\\nDatatype of 'User Location' column:\", data_type)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.746014Z","iopub.execute_input":"2023-11-08T11:16:56.746318Z","iopub.status.idle":"2023-11-08T11:16:56.759574Z","shell.execute_reply.started":"2023-11-08T11:16:56.746292Z","shell.execute_reply":"2023-11-08T11:16:56.757238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Count non-null values in the \"Tweet Coordinates\" column\nnon_null_count = unrest_tweets2[\"User Location\"].count()\n\n# Display the count\nprint(\"Number of non-null values in 'User Location':\", non_null_count)\n\n# Filter rows with non-null values in the \"User Location\" column\nnon_null_user_location_df = unrest_tweets2[unrest_tweets2[\"User Location\"].notnull()]\n\n# Check the size (number of rows and columns)\nnum_rows, num_columns = non_null_user_location_df.shape\nprint(\"Number of rows:\", num_rows)\nprint(\"Number of columns:\", num_columns)\n\n# Check for null values in the \"User Location\" column\nnull_values_count = non_null_user_location_df[\"User Location\"].isnull().sum()\nprint(\"Number of null values in 'User Location' column:\", null_values_count)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.761131Z","iopub.execute_input":"2023-11-08T11:16:56.762232Z","iopub.status.idle":"2023-11-08T11:16:56.776875Z","shell.execute_reply.started":"2023-11-08T11:16:56.762189Z","shell.execute_reply":"2023-11-08T11:16:56.775729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Test code for API to resolve locations into longitude and latitute:**","metadata":{}},{"cell_type":"code","source":"# import os\n# import pandas as pd\n# from geopy.geocoders import Nominatim\n\n# # Define the geocoding function\n# def geocode_location(location):\n#     geolocator = Nominatim(user_agent=\"Investigation Project\")\n#     try:\n#         location = geolocator.geocode(location)\n#         return (location.longitude, location.latitude)\n#     except AttributeError:\n#         return None\n\n# # Test the geocoding function with a sample location\n# sample_location = \"South Africa, Cape Town\"\n\n# # Call the geocoding function and print the result\n# result = geocode_location(sample_location)\n\n# if result is not None:\n#     longitude, latitude = result\n#     print(f\"Longitude: {longitude}, Latitude: {latitude}\")\n\n#     # Create a DataFrame from the geocoding result\n#     data = {'Location': [sample_location], 'Longitude': [longitude], 'Latitude': [latitude]}\n#     df = pd.DataFrame(data)\n\n#     # Define the output directory\n#     output_dir = '/kaggle/working/'\n\n#     # Save the DataFrame to a CSV file in the output directory\n#     output_filename = 'geocoding_result.csv'\n#     output_filepath = os.path.join(output_dir, output_filename)\n#     df.to_csv(output_filepath, index=False)\n\n#     print(f\"Geocoding result saved to {output_filepath}\")\n# else:\n#     print(\"Location not found or there was an issue with geocoding.\")\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.778755Z","iopub.execute_input":"2023-11-08T11:16:56.779787Z","iopub.status.idle":"2023-11-08T11:16:56.785042Z","shell.execute_reply.started":"2023-11-08T11:16:56.779745Z","shell.execute_reply":"2023-11-08T11:16:56.784151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The following code is used to query the Nominatim API to resolve the locations to longitudes and latitude for plotting. However, within the data, errors arose from the queries, so the data has been split into blocks to isolate which part of the data is corrupted/producing the query error. Each block of the overall data is stored in a csv which in turn will be combined.**","metadata":{}},{"cell_type":"code","source":"# import os\n# import pandas as pd\n# from geopy.geocoders import Nominatim\n# from geopy.exc import GeocoderQueryError\n\n# def geocode_and_save_block(user_locations, block_number, batch_size=100):\n#     # Define a function to geocode user locations to latitude and longitude\n#     def geocode_location_batch(locations, batch_size=batch_size):\n#         geolocator = Nominatim(user_agent=\"SA Tweets from Social Unrest\")\n#         total_locations = len(locations)\n#         geocoded_data = []\n        \n#         # Define a counter to keep track of successful geocoded locations\n#         geocoded_count = 0\n\n#         for i in range(0, total_locations, batch_size):\n#             batch_locations = locations[i:i + batch_size]\n#             batch_results = []\n\n#             for location in batch_locations:\n#                 try:\n#                     result = geolocator.geocode(location, timeout=10)  # Increase timeout\n#                     if result:\n#                         batch_results.append((location, result.longitude, result.latitude))\n#                         geocoded_count += 1\n#                 except (AttributeError, GeocoderQueryError) as e:\n#                     print(f\"Error geocoding location: {location} - {str(e)}\")\n\n#             geocoded_data.extend(batch_results)\n#             print(f\"Geocoded {geocoded_count} of {total_locations} locations.\")\n\n#         return geocoded_data\n\n#     # Split the data into four blocks\n#     total_data = len(user_locations)\n#     block_size = total_data // 4  # Divide the data into 4 equal blocks\n\n#     if block_number not in [1, 2, 3, 4]:\n#         raise ValueError(\"Block number should be 1, 2, 3, or 4.\")\n\n#     # Choose the specified block\n#     block = user_locations[(block_number - 1) * block_size:block_number * block_size]\n\n#     # Batch geocode the specified block of user locations\n#     geocoded_results = geocode_location_batch(block)\n\n#     # Create a DataFrame from the geocoded results\n#     geocoded_df = pd.DataFrame(geocoded_results, columns=[\"User Location\", \"Longitude\", \"Latitude\"])\n\n#     # Define the output CSV file path\n#     output_directory = '/kaggle/working/'\n#     output_filename = f'block{block_number}_geocoding_data.csv'\n#     output_filepath = os.path.join(output_directory, output_filename)\n\n#     # Save the geocoded data to a CSV file\n#     geocoded_df.to_csv(output_filepath, index=False)\n#     print(f\"Geocoded data saved to {output_filepath}\")\n\n#     print(f\"Geocoding completed successfully for block {block_number}.\")\n\n# # Extract the 'User Location' column as a list\n# user_locations = non_null_user_location_df[\"User Location\"].tolist()\n\n# # Specify which block to geocode and save (e.g., block_number=1)\n# geocode_and_save_block(user_locations, block_number=1)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.788866Z","iopub.execute_input":"2023-11-08T11:16:56.789447Z","iopub.status.idle":"2023-11-08T11:16:56.800859Z","shell.execute_reply.started":"2023-11-08T11:16:56.789409Z","shell.execute_reply":"2023-11-08T11:16:56.799718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport os\n\n# Define the directory path where the CSV files are located\ndirectory = '/kaggle/input/all-geocoded-data/'\n\n# Define the file names of the CSV files\nfile_names = [\n    'block1_geocoding_data.csv',\n    'block2_geocoding_data.csv',\n    'block3_geocoding_data.csv',\n    'block4_geocoding_data.csv'\n]\n\n# Initialize an empty list to store DataFrames\ndata_frames = []\n\n# Define a function to preprocess the CSV data\ndef preprocess_csv(file_path):\n    with open(file_path, 'r') as file:\n        lines = file.readlines()\n\n    # Initialize lists to store valid longitude and latitude values\n    valid_longitude = []\n    valid_latitude = []\n\n    for line in lines[1:]:\n        values = line.strip().split(',')\n        # Check if there are at least 2 values (longitude and latitude)\n        if len(values) >= 2:\n            # Attempt to convert longitude and latitude to floats\n            try:\n                longitude = float(values[-2])\n                latitude = float(values[-1])\n                valid_longitude.append(longitude)\n                valid_latitude.append(latitude)\n            except ValueError:\n                # Skip lines with invalid longitude or latitude values\n                continue\n\n    # Create a DataFrame from the valid longitude and latitude values\n    df = pd.DataFrame({'Longitude': valid_longitude, 'Latitude': valid_latitude})\n\n    return df\n\nfor file_name in file_names:\n    print(f\"Reading file: {file_name}\")\n    file_path = directory + file_name\n    # Preprocess the CSV file\n    df = preprocess_csv(file_path)\n    data_frames.append(df)\n\n# Concatenate the DataFrames vertically to combine them\ncombined_df = pd.concat(data_frames, ignore_index=True)\n\n# Display key features of the combined DataFrame\nprint(\"Head of the Combined DataFrame:\")\nprint(combined_df.head())\n\nprint(\"\\nNumber of Rows and Columns in the Combined DataFrame:\")\nprint(combined_df.shape)\n\n# Now, you have the combined DataFrame containing only valid longitude and latitude.","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.802157Z","iopub.execute_input":"2023-11-08T11:16:56.803218Z","iopub.status.idle":"2023-11-08T11:16:56.851326Z","shell.execute_reply.started":"2023-11-08T11:16:56.803169Z","shell.execute_reply":"2023-11-08T11:16:56.850534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import geopandas as gpd\nfrom shapely.geometry import Point\nimport matplotlib.pyplot as plt\n\n\n# Create Point geometries using Shapely\ngeometry = [Point(xy) for xy in zip(combined_df['Longitude'], combined_df['Latitude'])]\n\n# Create a GeoDataFrame\ncrs = 'EPSG:4326'  # Assuming the coordinates are in WGS 84\ngeo_df = gpd.GeoDataFrame(combined_df, crs=crs, geometry=geometry)\n\n# Download world shapefile from GeoPandas datasets (if not already downloaded)\n# gpd.datasets.get_path('naturalearth_lowres').to_csv(\"path_to_shapefile.zip\")\n \n# Load world shapefile\nworld = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\n\n# Create subplots\nfig, ax = plt.subplots(1, figsize=(16, 8), facecolor='lightblue')\n\n# Plot the world map with white color\nworld.plot(ax=ax, color='white', edgecolor='black')\n\n# Plot your geo_df data\ngeo_df.plot(ax=ax, markersize=1, color='m', marker='o')\n\n# Turn off the axis\nax.axis('off')\n\n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:56.852692Z","iopub.execute_input":"2023-11-08T11:16:56.852998Z","iopub.status.idle":"2023-11-08T11:16:58.930853Z","shell.execute_reply.started":"2023-11-08T11:16:56.85297Z","shell.execute_reply":"2023-11-08T11:16:58.929426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import geopandas as gpd\nfrom shapely.geometry import Point\nimport matplotlib.pyplot as plt\n\n\n# Create Point geometries using Shapely\ngeometry = [Point(xy) for xy in zip(combined_df['Longitude'], combined_df['Latitude'])]\n\n# Create a GeoDataFrame\ncrs = 'EPSG:4326'  # Assuming the coordinates are in WGS 84\ngeo_df = gpd.GeoDataFrame(combined_df, crs=crs, geometry=geometry)\n\n# Load South African provinces shapefile\nsa_provinces = gpd.read_file('/kaggle/input/natural-earth-provinces/ne_10m_admin_1_states_provinces.shp')\n\n# Filter the shapefile to get only South Africa\nsouth_africa = sa_provinces[sa_provinces['admin'] == 'South Africa']\n\n# Get the bounding box of South Africa\nsa_bbox = south_africa.bounds\n\n# Set the map extent to cover South Africa's bounding box\nfig, ax = plt.subplots(1, figsize=(16, 8), facecolor='white')\nax.set_xlim([sa_bbox.minx.min(), sa_bbox.maxx.max()])\nax.set_ylim([sa_bbox.miny.min(), sa_bbox.maxy.max()])\n\n# Plot the South African provinces\nsouth_africa.boundary.plot(ax=ax, linewidth=2, color='black')\n\n# Plot your geo_df data within the bounds of South Africa\ngeo_df.plot(ax=ax, markersize=1, color='m', marker='o')\n\n# Turn off the axis\nax.axis('off')\n\n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:16:58.932561Z","iopub.execute_input":"2023-11-08T11:16:58.932997Z","iopub.status.idle":"2023-11-08T11:17:25.719858Z","shell.execute_reply.started":"2023-11-08T11:16:58.932956Z","shell.execute_reply":"2023-11-08T11:17:25.718553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import geopandas as gpd\nfrom shapely.geometry import Point\nimport matplotlib.pyplot as plt\n\n# Create Point geometries using Shapely\ngeometry = [Point(xy) for xy in zip(combined_df['Longitude'], combined_df['Latitude'])]\n\n# Create a GeoDataFrame\ncrs = 'EPSG:4326'  # Assuming the coordinates are in WGS 84\ngeo_df = gpd.GeoDataFrame(combined_df, crs=crs, geometry=geometry)\n\n# Load South African provinces shapefile\nsa_provinces = gpd.read_file('/kaggle/input/natural-earth-provinces/ne_10m_admin_1_states_provinces.shp')\n\n# Filter the shapefile to get only South Africa\nsouth_africa = sa_provinces[sa_provinces['admin'] == 'South Africa']\n\n# Get the bounding box of South Africa\nsa_bbox = south_africa.bounds\n# Set the map extent to cover South Africa's bounding box\nfig, ax = plt.subplots(1, figsize=(16, 8), facecolor='white')\nax.set_xlim([sa_bbox.minx.min(), sa_bbox.maxx.max()])\nax.set_ylim([sa_bbox.miny.min(), sa_bbox.maxy.max()])\n\n# Plot the South African provinces with lighter border lines\nsouth_africa.boundary.plot(ax=ax, linewidth=1, color='black', edgecolor='gray')  # Adjust linewidth and edgecolor\n\n# Plot your geo_df data within the bounds of South Africa\ngeo_df.plot(ax=ax, markersize=1, color='b', marker='o')\n\n# Turn off the axis\nax.axis('off')\n\n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:25.721969Z","iopub.execute_input":"2023-11-08T11:17:25.722401Z","iopub.status.idle":"2023-11-08T11:17:52.908116Z","shell.execute_reply.started":"2023-11-08T11:17:25.722361Z","shell.execute_reply":"2023-11-08T11:17:52.906927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import geopandas as gpd\nfrom shapely.geometry import Point\nimport matplotlib.pyplot as plt\n\n# Your code to create geo_df and set up the world map goes here...\n\n# Count the number of plotted points for the entire world\nnum_points_world = len(geo_df)\n\n# Filter geo_df to include only points within the bounds of South Africa\ngeo_df_sa = geo_df.cx[sa_bbox.minx.min():sa_bbox.maxx.max(), sa_bbox.miny.min():sa_bbox.maxy.max()]\n\n# Count the number of plotted points for South Africa\nnum_points_sa = len(geo_df_sa)\n\n# Show the counts\nprint(f\"Number of plotted points in the entire world: {num_points_world}\")\nprint(f\"Number of plotted points in South Africa: {num_points_sa}\")\n\n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:52.909552Z","iopub.execute_input":"2023-11-08T11:17:52.91043Z","iopub.status.idle":"2023-11-08T11:17:52.944377Z","shell.execute_reply.started":"2023-11-08T11:17:52.910396Z","shell.execute_reply":"2023-11-08T11:17:52.942408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-two\"></a>\n# **2. GDELT Analysis**\n##  **Global Database of Events, Language, and Tone (GDELT)**\n\n<a id=\"section-two-a\"></a>\nThe Global Database of Events, Language, and Tone (GDELT) is a comprehensive and continuously updated dataset that monitors and records various global events, news articles, and media sources from around the world. GDELT's primary purpose is to provide a vast repository of structured data that researchers, analysts, and data scientists can use to analyze and gain insights into global events, trends, and sentiments.\n​\nKey features of GDELT include:\n​\n- **Event Data**: GDELT captures a wide range of events, including political, economic, social, and cultural events, across different countries and regions.\n​\n- **Media Monitoring**: GDELT scans thousands of news articles, broadcasts, and online sources in multiple languages to extract valuable information.\n​\n- **Sentiment Analysis**: It includes sentiment analysis and tone indicators, helping to understand the emotional context of news and events.\n​\n- **Temporal Coverage**: GDELT's data goes back several decades, allowing users to explore historical trends and patterns.\n​\n- **Geospatial Information**: The dataset includes geospatial information, enabling the mapping of events and their locations.\n​\nhttps://www.gdeltproject.org/\n​\n\nThe Global Database of Events, Language, and Tone (GDELT) is a comprehensive and continuously updated dataset that monitors and records various global events, news articles, and media sources from around the world. GDELT's primary purpose is to provide a vast repository of structured data that researchers, analysts, and data scientists can use to analyze and gain insights into global events, trends, and sentiments.\n                                                                                                                                                                                                               ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-two-b\"></a>\n## **GDELT Exploratory Data Analysis (EDA)**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n \n\n# Define the directory and file name\ndirectory = \"south-africa-gdelts-riots-2021\"\nfile_name = \"Query_riots_2021_test6_allColumns.csv\"\n\n \n\n# Create the full file path\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\n\n \n\n# Load the dataset into a Pandas DataFrame\ntry:\n    df = pd.read_csv(file_path)\n    \n    # Display the main specifications of the dataset\n    print(\"Dataset Specifications:\")\n    print(f\"File Path: {file_path}\")\n    print(f\"Number of Rows: {len(df)}\")\n    print(f\"Number of Columns: {len(df.columns)}\")\n    print(\"Column Names:\")\n    for column in df.columns:\n        print(f\" - {column}\")\n    print(\"Data Types:\")\n    for column, dtype in df.dtypes.items():\n        print(f\" - {column}: {dtype}\")\n#     print(\"Summary Statistics:\")\n#     print(df.describe())\nexcept FileNotFoundError:\n    print(f\"File '{file_path}' not found.\")\nexcept Exception as e:\n    print(f\"An error occurred: {str(e)}\")","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:52.946155Z","iopub.execute_input":"2023-11-08T11:17:52.946656Z","iopub.status.idle":"2023-11-08T11:17:53.012593Z","shell.execute_reply.started":"2023-11-08T11:17:52.946611Z","shell.execute_reply":"2023-11-08T11:17:53.011691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-two-c\"></a>\n## **Total Number of Mentions per day**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n# Load the dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df['SQLDATE'].value_counts().sort_index()\n\n# Create a bar graph\nplt.figure(figsize=(12, 6))\ndate_counts.plot(kind='bar', color='skyblue')\nplt.title('Number of articles per day')\nplt.xlabel('DATE')\nplt.ylabel('Count')\nplt.xticks(rotation=90)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:53.014521Z","iopub.execute_input":"2023-11-08T11:17:53.014915Z","iopub.status.idle":"2023-11-08T11:17:53.455382Z","shell.execute_reply.started":"2023-11-08T11:17:53.014885Z","shell.execute_reply":"2023-11-08T11:17:53.454561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n# Load the dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df['SQLDATE'].value_counts().sort_index()\n\n# Create a line graph\nplt.figure(figsize=(12, 6))\ndate_counts.plot(kind='line', color='skyblue')\nplt.title('Number of articles per day')\nplt.xlabel('DATE')\nplt.ylabel('Count')\nplt.xticks(rotation=90)\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:53.456956Z","iopub.execute_input":"2023-11-08T11:17:53.457608Z","iopub.status.idle":"2023-11-08T11:17:53.785631Z","shell.execute_reply.started":"2023-11-08T11:17:53.457568Z","shell.execute_reply":"2023-11-08T11:17:53.784546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Number of Mentions of the Article per day**\n\n**Definition of NumMentions:**\nNumMentions=\t(NULLABLE;\tINTEGER)\tThis is the total number of mentions of this event across all source documents during the 15 minute update in which it was first seen. Multiple references to an event within a single document also contribute to this count. This can be used as a method of assessing the “importance” of an event: the more discussion of that event, the more likely it is to be significant. The total universe of source documents and the density of events within them vary over time, so it is recommended that this field be normalized by the average or other measure of the universe of events during the time period of interest. This field is actually a composite score of the total number of raw mentions and the number of mentions extracted from reprocessed versions of each article (see the discussion for the Mentions table). NOTE: this field refers only to the first news report to mention an event and is not updated if the event is found in a different context in other news reports. It is included for legacy purposes – for more precise information on the positioning of an event, see the Mentions table.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n# Load the dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and sum the NumMentions for each date\ndate_mentions = df.groupby('SQLDATE')['NumMentions'].sum()\n\n# Create a bar graph\nplt.figure(figsize=(12, 6))\ndate_mentions.plot(kind='bar', color='skyblue')\nplt.title('Total Number of NumMentions per Day')\nplt.xlabel('SQLDATE')\nplt.ylabel('Total NumMentions')\nplt.xticks(rotation=90)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:53.787062Z","iopub.execute_input":"2023-11-08T11:17:53.787525Z","iopub.status.idle":"2023-11-08T11:17:54.149123Z","shell.execute_reply.started":"2023-11-08T11:17:53.787486Z","shell.execute_reply":"2023-11-08T11:17:54.148022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-two-d\"></a>\n## **GDELT Sentiment Analysis**","metadata":{}},{"cell_type":"markdown","source":"**Distribution of sentiment of articles**\nThis uses the the column title AvgTone to gauge the sentiment.\n\n**Definition of AvgTone:**\nAvgTone=\t(NULLABLE;\tFLOAT)\tThis is the average “tone” of all documents containing one or more mentions of this event during the 15 minute update in which it was first seen. The score ranges from -100 (extremely negative) to +100 (extremely positive). Common values range between -10 and +10, with 0 indicating neutral. This can be used as a method of filtering the “context” of events as a subtle measure of the importance of an event and as a proxy for the “impact” of that event. For example, a riot event with a slightly negative average tone is likely to have been a minor occurrence, whereas if it had an extremely negative average tone, it suggests a far more serious occurrence. A riot with a positive score likely suggests a very minor occurrence described in the context of a more positive narrative (such as a report of an attack occurring in a discussion of improving conditions on the ground in a country and how the number of attacks per day has been greatly reduced). NOTE: this field refers only to the first news report to mention an event and is not updated if the event is found in a different context in other news reports. It is included for legacy purposes – for more precise information on the positioning of an event, see the Mentions table. NOTE: this provides only a basic tonal assessment of an article and it is recommended that users interested in emotional measures use the Mentions and Global Knowledge Graph tables to merge the complete set of 2,300 emotions and themes from the GKG GCAM system into their analysis of event records. (Source: GDELT database)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n# Load the dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Create a histogram of AvgTone with a bin width of 1 unit\nbin_width = 1.0\nplt.figure(figsize=(12, 6))\nplt.hist(df['AvgTone'], bins=int((df['AvgTone'].max() - df['AvgTone'].min()) / bin_width), color='skyblue')\nplt.title('Average sentiment of articles')\nplt.xlabel('Average Sentiment')\nplt.ylabel('Number of Articles')\nplt.grid(axis='y', linestyle='--', alpha=0.7)\nplt.tight_layout()\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:54.15065Z","iopub.execute_input":"2023-11-08T11:17:54.151001Z","iopub.status.idle":"2023-11-08T11:17:54.651933Z","shell.execute_reply.started":"2023-11-08T11:17:54.150964Z","shell.execute_reply":"2023-11-08T11:17:54.650707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\nstart_date = ''\nend_date = ''\n\n# Load the dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path, parse_dates=['SQLDATE'])\n\n# Group the data by date and calculate the average sentiment (AvgTone) for each day\naverage_sentiment_per_day = df.groupby('SQLDATE')['AvgTone'].mean()\n\n# Plot the average sentiment over time\nplt.figure(figsize=(12, 6))\nplt.plot(average_sentiment_per_day.index, average_sentiment_per_day.values, marker='o', linestyle='-')\nplt.title('Average Sentiment per Day Over Time')\nplt.xlabel('Date')\nplt.ylabel('Average Sentiment (AvgTone)')\nplt.grid(True)\n\n# Set the date range from '2020-10-15' to '2020-11-08'\ndate_range = pd.date_range(start='2021-07-09', end='2021-07-21', freq='D')\nall_dates = date_range.tolist()\n\n# # Set the x-axis ticks to match the SQLDATE values\n# dates_to_highlight = [\"2020-11-03\", \"2020-11-07\", \"2020-10-22\"]  # Add your desired dates here\n# title_of_dates = [\"Election Day\", \"Highest Tweets\", \"Final Presidential Debate\"]\n# colors = [\"red\", \"green\", \"yellow\"]\n\n# # Calculate the positions for evenly spaced x-axis ticks\n# x_positions = [average_sentiment_per_day.index.get_loc(date) for date in dates_to_highlight]\n\n# for i, date in enumerate(dates_to_highlight):\n#     title = title_of_dates[i]\n#     color = colors[i]\n#     date_current = dates_to_highlight[i]\n#     position = pd.Timestamp(date_current)\n#     plt.axvline(x=position, color=color, linestyle='--', label=title)\n\n# Set the x-axis ticks to be all_dates and their labels\nplt.xticks(all_dates, [date.strftime('%Y-%m-%d') for date in all_dates], rotation=45)\n\nplt.xticks(rotation=80)\n# Show the plot\nplt.tight_layout()\nplt.legend()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:54.653556Z","iopub.execute_input":"2023-11-08T11:17:54.654006Z","iopub.status.idle":"2023-11-08T11:17:55.106012Z","shell.execute_reply.started":"2023-11-08T11:17:54.653966Z","shell.execute_reply":"2023-11-08T11:17:55.104821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-three\"></a>\n# **3. Google Trends Data Analysis**","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-three-c\"></a>\n## **Google Trends Description**","metadata":{}},{"cell_type":"markdown","source":"For the Google Trends analysis conducted, we focused on the search interest of three key keywords:\n\n1. **Looting**\n2. **State of Emergency**\n3. **Protest**\n\nThese keywords were selected to examine the search interest and popularity trends surrounding the 2023, Social Unrest in South Africa.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-three-b\"></a>\n## **Web Search**\n\n[Google Trends Analysis: Looting,State of Emergency, Protest](https://trends.google.com/trends/explore?date=2021-07-09%202021-07-21&geo=ZA&q=Looting,State%20of%20Emergency,Protest)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\n\n# Create a string with the CSV data\ncsv_data = \"\"\"Day,Looting_web,State of emergency_web,Protest_web\n2021-07-09,<1,<1,5\n2021-07-10,1,0,5\n2021-07-11,7,<1,11\n2021-07-12,68,12,23\n2021-07-13,100,16,13\n2021-07-14,73,25,7\n2021-07-15,44,5,2\n2021-07-16,28,3,2\n2021-07-17,17,1,1\n2021-07-18,13,1,2\n2021-07-19,11,<1,1\n2021-07-20,8,<1,<1\n2021-07-21,6,<1,1\"\"\"\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Replace \"<1\" and \"<1,5\" values with appropriate numerical values\ndf['Looting_web'] = df['Looting_web'].replace(['<1', '<1,5'], [0.5, 1])\ndf['State of emergency_web'] = df['State of emergency_web'].replace(['<1', '<1,5'], [0.5, 1])\ndf['Protest_web'] = df['Protest_web'].replace(['<1', '<1,5'], [0.5, 1])\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Convert \"Protest_web\" column to numeric\ndf['Protest_web'] = pd.to_numeric(df['Protest_web'], errors='coerce')\n\nlooting_web_df = pd.DataFrame({'Day': df['Day'], 'Looting Web': df['Looting_web']})\nstate_of_emergency_web_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency Web': df['State of emergency_web']})\nprotest_web_df = pd.DataFrame({'Day': df['Day'], 'Protest Web': df['Protest_web']})\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the data\nplt.plot(df['Day'], df['Looting_web'], label='\"Looting\"', marker='o')\nplt.plot(df['Day'], df['State of emergency_web'], label='\"State of Emergency\"', marker='o')\nplt.plot(df['Day'], df['Protest_web'], label='\"Protest\"', marker='o')\n\n# Set y-axis range from 0 to 105\nplt.ylim(0, 105)\n\n# Set x-axis ticks to match the dates\nplt.xticks(df['Day'], rotation=45)\n\n# Add vertical lines for specific dates\ndates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\ntitle_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\ncolors = [\"red\", \"green\"]\n\nfor date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n    ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n# Merge both legends into a single legend\nlegend = ax.legend(loc='upper left')\n\n# Add labels and title\nplt.xlabel('Date')\nplt.ylabel('Search Interest')\nplt.title('Web Search Trends Data (July 9, 2021 - July 21, 2021)')\n\n# Show the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:55.107807Z","iopub.execute_input":"2023-11-08T11:17:55.108121Z","iopub.status.idle":"2023-11-08T11:17:55.607041Z","shell.execute_reply.started":"2023-11-08T11:17:55.108093Z","shell.execute_reply":"2023-11-08T11:17:55.605852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-three-c\"></a>\n## **News Search**\n\n[Google Trends Analysis:Looting, State of Emergency, Protest](https://trends.google.com/trends/explore?date=2021-07-09%202021-07-21&geo=ZA&gprop=news&q=Looting,State%20of%20Emergency,Protest) ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\n\n# Create a string with the CSV data\ncsv_data = \"\"\"Day,Looting_news,State of emergency_news,Protest_news\n2021-07-09,0,0,9\n2021-07-10,0,0,12\n2021-07-11,0,6,15\n2021-07-12,55,23,25\n2021-07-13,100,21,25\n2021-07-14,54,12,0\n2021-07-15,30,0,39\n2021-07-16,22,0,0\n2021-07-17,0,0,7\n2021-07-18,0,0,0\n2021-07-19,13,0,0\n2021-07-20,11,10,0\n2021-07-21,12,0,14\n\"\"\"\n\n\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Create DataFrames for each curve\nlooting_news_df = pd.DataFrame({'Day': df['Day'], 'Looting News': df['Looting_news']})\nstate_of_emergency_news_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency News': df['State of emergency_news']})\nprotest_news_df = pd.DataFrame({'Day': df['Day'], 'Protest News': df['Protest_news']})\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the data\nplt.plot(df['Day'], df['Looting_news'], label='Looting', marker='o')\nplt.plot(df['Day'], df['State of emergency_news'], label='State of Emergency', marker='o')\nplt.plot(df['Day'], df['Protest_news'], label='Protest', marker='o')\n\n# Set y-axis range from 0 to the maximum value in the dataset\nplt.ylim(0, df[['Looting_news', 'State of emergency_news', 'Protest_news']].max().max() + 5)\n\n# Set x-axis ticks to match the dates\nplt.xticks(df['Day'], rotation=45)\n\n\n# Add vertical lines for specific dates\ndates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\ntitle_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\ncolors = [\"red\", \"green\"]\n\nfor date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n    ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n# Merge both legends into a single legend\nlegend = ax.legend(loc='upper left')\n\n# Add labels and title\nplt.xlabel('Date')\nplt.ylabel('Search Interest')\nplt.title('News Search Trends Data (July 9, 2021 - July 21, 2021)')\n\n# Show the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:55.608784Z","iopub.execute_input":"2023-11-08T11:17:55.609091Z","iopub.status.idle":"2023-11-08T11:17:56.095824Z","shell.execute_reply.started":"2023-11-08T11:17:55.609064Z","shell.execute_reply":"2023-11-08T11:17:56.094636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-four\"></a>\n# **4. Pattern of Interest Comparisons**","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-four-a\"></a>\n## **Comparison of Number of Tweets and GDELT Articles per Day**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Assuming unrest_tweets2 is your DataFrame with the 'date' column (format from (1))\n\n# Load the dataset from (2)\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df['SQLDATE'].value_counts().sort_index()\n\n# Convert SQLDATE to the same datetime format as in (1)\ndate_counts.index = pd.to_datetime(date_counts.index, format='%Y%m%d')\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the first dataset (from (1))\nax.plot(tweets_per_day.index, tweets_per_day.values, marker='o', label='Number of Tweets per Day', color='skyblue')\n\n# Plot the second dataset (converted from SQLDATE)\nax.plot(date_counts.index, date_counts.values, marker='o', label='Number of GDELT Articles per Day', color='orange')\n\n# Set the x-axis format to display only the date\ndate_formatter = DateFormatter('%Y-%m-%d')\nax.xaxis.set_major_formatter(date_formatter)\n\n# Set the dates starting from the beginning of the event:\nspecific_dates = [pd.Timestamp('2021-07-09')]\ndate_range = pd.date_range(start='2021-07-10', end='2021-07-18', freq='D')\nall_dates = specific_dates + date_range.tolist()\n\nax.set_xticks(all_dates)\n\n# Rotate the x-axis labels for better visibility\nplt.xticks(rotation=90)\n\n# Set labels and title\nax.set_xlabel('Date')\nax.set_ylabel('Count')\nax.set_title('Number of Tweets and GDELT Articles per Day')\n\n# Add a legend\nax.legend()\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:56.097445Z","iopub.execute_input":"2023-11-08T11:17:56.09884Z","iopub.status.idle":"2023-11-08T11:17:56.571651Z","shell.execute_reply.started":"2023-11-08T11:17:56.098787Z","shell.execute_reply":"2023-11-08T11:17:56.570547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Assuming unrest_tweets2 is your DataFrame with the 'date' column (format from (1))\n\n# Load the dataset from (2)\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df['SQLDATE'].value_counts().sort_index()\n\n# Convert SQLDATE to the same datetime format as in (1)\ndate_counts.index = pd.to_datetime(date_counts.index, format='%Y%m%d')\n\n# Normalize the first dataset (from (1)) to the range [0, 1]\nmax_value_tweets = tweets_per_day.max()\nnormalized_tweets = tweets_per_day / max_value_tweets\n\n# Normalize the second dataset (converted from SQLDATE) to the range [0, 1]\nmax_value_articles = date_counts.max()\nnormalized_date_counts = date_counts / max_value_articles\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the first dataset (normalized from (1)) on the left y-axis\nax.plot(tweets_per_day.index, normalized_tweets.values, marker='o', label='Normalized Number of Tweets per Day', color='skyblue')\n\n# Plot the second dataset (normalized) on the same y-axis\nax.plot(date_counts.index, normalized_date_counts.values, marker='o', label='Normalized Number of GDELT Articles per Day', color='orange')\n\n# Ensure that all dates are displayed on the x-axis\ndate_range = pd.date_range(start='2021-07-09', end='2021-07-21', freq='D')\nax.set_xticks(date_range)\n\nfrom datetime import datetime, timedelta\n\n# Initialize an empty list to store weekend dates\nweekend_dates = []\n\n# Iterate through all dates and check if they are weekends\nfor i in range(len(date_range)):\n    date = date_range[i]\n    if i == 0 and date.dayofweek == 6:\n        weekend_dates.append([date,date + timedelta(days=1)])\n    elif date.dayofweek == 5:\n        weekend_dates.append([date,date + timedelta(days=1)])\n    elif date.dayofweek == 6:\n        weekend_dates[-1][1]= date + timedelta(days=1)\n    \n\n# Print the list of weekend dates\nprint(\"Weekend Days:\")\nprint(weekend_dates)\n\nfor weekend in weekend_dates:\n    # Add shaded region for the period between '2020-11-03' and '2020-11-07'\n    ax.axvspan(weekend[0], weekend[1], color='gray', alpha=0.1, label='Weekend')\n\n# Rotate the x-axis labels for better visibility\nax.tick_params(axis='x', rotation=90)\n\n# Add vertical lines for specific dates\ndates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\ntitle_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\ncolors = [\"red\", \"green\"]\n\nfor date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n    ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n\n# Set labels and title\nax.set_xlabel('Date')\nax.set_ylabel('Normalized Count')\nax.set_title('Comparison of Normalized Tweets and Normalized GDELT Articles per Day')\n\n# Add a legend for both datasets\nax.legend(loc='upper right')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:56.573382Z","iopub.execute_input":"2023-11-08T11:17:56.574068Z","iopub.status.idle":"2023-11-08T11:17:57.122969Z","shell.execute_reply.started":"2023-11-08T11:17:56.574029Z","shell.execute_reply":"2023-11-08T11:17:57.121551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-four-b\"></a>\n## **Comparison of Normalized Tweets and Normalized Google Trends Web Search Data**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\n\n# Create a string with the CSV data and manually replace \"<1\" with \"0.5\"\ncsv_data = \"\"\"Day,Looting_web,State of emergency_web,Protest_web\n2021-07-09,0.5,0.5,5\n2021-07-10,1,0,5\n2021-07-11,7,0.5,11\n2021-07-12,68,12,23\n2021-07-13,100,16,13\n2021-07-14,73,25,7\n2021-07-15,44,5,2\n2021-07-16,28,3,2\n2021-07-17,17,1,1\n2021-07-18,13,1,2\n2021-07-19,11,0.5,1\n2021-07-20,8,0.5,0.5\n2021-07-21,6,0.5,1\"\"\"\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Normalize all dataframes to their respective max values\ndf['Looting_web'] = df['Looting_web'] / df['Looting_web'].max()\ndf['State of emergency_web'] = df['State of emergency_web'] / df['State of emergency_web'].max()\ndf['Protest_web'] = df['Protest_web'] / df['Protest_web'].max()\n\nlooting_web_df = pd.DataFrame({'Day': df['Day'], 'Looting Web (Normalized)': df['Looting_web']})\nstate_of_emergency_web_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency Web (Normalized)': df['State of emergency_web']})\nprotest_web_df = pd.DataFrame({'Day': df['Day'], 'Protest Web (Normalized)': df['Protest_web']})\n\n# Assuming earthquake_tweets is your DataFrame with the 'date' column\nunrest_tweets2['date'] = pd.to_datetime(unrest_tweets2[\"Tweet Created At\"])  # Convert the 'date' column to datetime\n\n# Resample the data by day and count the number of tweets\ntweets_per_day = unrest_tweets2.resample('D', on='date').size()\n\n# Normalize the tweets_per_day data to its max value\ntweets_per_day_normalized = tweets_per_day / tweets_per_day.max()\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the normalized data\nplt.plot(df['Day'], df['Looting_web'], label='Looting Web (Normalized)', marker='o')\nplt.plot(df['Day'], df['State of emergency_web'], label='State of Emergency Web (Normalized)', marker='o')\nplt.plot(df['Day'], df['Protest_web'], label='Protest Web (Normalized)', marker='o')\nplt.plot(tweets_per_day_normalized.index, tweets_per_day_normalized.values, label='Tweets per Day (Normalized)', marker='o')\n\n# Set y-axis range from 0 to 1 (normalized range)\nplt.ylim(0, 1.1)\n\n# Set x-axis ticks to match the dates\nplt.xticks(df['Day'], rotation=45)\n\n# Add vertical lines for specific dates\ndates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\ntitle_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\ncolors = [\"red\", \"green\"]\n\nfor date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n    ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n# Merge all legends into a single legend\nlegend = ax.legend(loc='upper right')\n\n# Add labels and title\nplt.xlabel('Date')\nplt.ylabel('Normalized Values')\nplt.title('Normalized Data (Web Search Trends and Tweets)')\n\n# Show the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:57.125105Z","iopub.execute_input":"2023-11-08T11:17:57.125577Z","iopub.status.idle":"2023-11-08T11:17:58.628989Z","shell.execute_reply.started":"2023-11-08T11:17:57.125532Z","shell.execute_reply":"2023-11-08T11:17:58.627847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\nimport ipywidgets as widgets\nfrom ipywidgets import interact\n\n# ... (Previous code to load and preprocess data)\n# Create a string with the CSV data and manually replace \"<1\" with \"0.5\"\ncsv_data = \"\"\"Day,Looting_web,State of emergency_web,Protest_web\n2021-07-09,0.5,0.5,5\n2021-07-10,1,0,5\n2021-07-11,7,0.5,11\n2021-07-12,68,12,23\n2021-07-13,100,16,13\n2021-07-14,73,25,7\n2021-07-15,44,5,2\n2021-07-16,28,3,2\n2021-07-17,17,1,1\n2021-07-18,13,1,2\n2021-07-19,11,0.5,1\n2021-07-20,8,0.5,0.5\n2021-07-21,6,0.5,1\"\"\"\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Normalize all dataframes to their respective max values\ndf['Looting_web'] = df['Looting_web'] / df['Looting_web'].max()\ndf['State of emergency_web'] = df['State of emergency_web'] / df['State of emergency_web'].max()\ndf['Protest_web'] = df['Protest_web'] / df['Protest_web'].max()\n\nlooting_web_df = pd.DataFrame({'Day': df['Day'], 'Looting Web (Normalized)': df['Looting_web']})\nstate_of_emergency_web_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency Web (Normalized)': df['State of emergency_web']})\nprotest_web_df = pd.DataFrame({'Day': df['Day'], 'Protest Web (Normalized)': df['Protest_web']})\n\n# Create a function to update the plot based on the selected datasets\ndef update_plot(show_tweets, show_looting, show_state, show_protest):\n    fig, ax = plt.subplots(figsize=(12, 6))\n\n    if show_tweets:\n        ax.plot(tweets_per_day.index, tweets_per_day_normalized, marker='o', label='Tweets per Day (Normalized)', color='skyblue')\n    if show_looting:\n        ax.plot(df['Day'], df['Looting_web'], marker='o', label='Looting Web (Normalized)', color='orange')\n    if show_state:\n        ax.plot(df['Day'], df['State of emergency_web'], marker='o', label='State of Emergency Web (Normalized)', color='green')\n    if show_protest:\n        ax.plot(df['Day'], df['Protest_web'], marker='o', label='Protest Web (Normalized)', color='red')\n\n    # Add vertical lines for specific dates\n    dates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\n    title_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\n    colors = [\"purple\", \"yellow\"]\n\n    for date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n        ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n    # Merge all legends into a single legend\n    legend = ax.legend(loc='upper left')\n\n    # Ensure that all dates are displayed on the x-axis\n    date_range = pd.date_range(start='2021-07-09', end='2021-07-21', freq='D')\n    ax.set_xticks(date_range)\n\n    # Rotate the x-axis labels for better visibility\n    ax.tick_params(axis='x', rotation=45)\n\n    # Set labels and title\n    ax.set_xlabel('Date')\n    ax.set_ylabel('Normalized Values')\n    ax.set_title('Normalized Data (Web Search Trends and Tweets')\n\n    plt.tight_layout()\n    plt.show()\n\n# Create checkboxes to select the datasets\nshow_tweets_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show Tweets per day',\n    disabled=False\n)\n\nshow_looting_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show Looting Web Signature',\n    disabled=False\n)\n\nshow_state_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show State of Emergency Web Signature',\n    disabled=False\n)\n\nshow_protest_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show \"Protest\" Web Signature',\n    disabled=False\n)\n\n# Create an interactive plot\ninteract(update_plot, show_tweets=show_tweets_checkbox, show_looting=show_looting_checkbox,\n         show_state=show_state_checkbox, show_protest=show_protest_checkbox)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:58.630932Z","iopub.execute_input":"2023-11-08T11:17:58.631567Z","iopub.status.idle":"2023-11-08T11:17:59.332899Z","shell.execute_reply.started":"2023-11-08T11:17:58.631522Z","shell.execute_reply":"2023-11-08T11:17:59.331519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-four-c\"></a>\n## **Comparison of Normalized GDELT Articles and Normalized Google Trends News Search Data**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\n\n# Create a string with the CSV data for the news search trends\ncsv_data = \"\"\"Day,Looting_news,State of emergency_news,Protest_news\n2021-07-09,0,0,9\n2021-07-10,0,0,12\n2021-07-11,0,6,15\n2021-07-12,55,23,25\n2021-07-13,100,21,25\n2021-07-14,54,12,0\n2021-07-15,30,0,39\n2021-07-16,22,0,0\n2021-07-17,0,0,7\n2021-07-18,0,0,0\n2021-07-19,13,0,0\n2021-07-20,11,10,0\n2021-07-21,12,0,14\n\"\"\"\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Create DataFrames for each curve\nlooting_news_df = pd.DataFrame({'Day': df['Day'], 'Looting News': df['Looting_news']})\nstate_of_emergency_news_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency News': df['State of emergency_news']})\nprotest_news_df = pd.DataFrame({'Day': df['Day'], 'Protest News': df['Protest_news']})\n\n# Normalize each DataFrame to its own max value\nlooting_news_df['Looting News'] = looting_news_df['Looting News'] / looting_news_df['Looting News'].max()\nstate_of_emergency_news_df['State of Emergency News'] = state_of_emergency_news_df['State of Emergency News'] / state_of_emergency_news_df['State of Emergency News'].max()\nprotest_news_df['Protest News'] = protest_news_df['Protest News'] / protest_news_df['Protest News'].max()\n\n# Load the GDELT dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf_gdelt = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df_gdelt['SQLDATE'].value_counts().sort_index()\n\n# Convert SQLDATE to the same datetime format as in the news search trends data\ndate_counts.index = pd.to_datetime(date_counts.index, format='%Y%m%d')\n\n# Normalize the GDELT data with its own max value\ndate_counts_normalized = date_counts / date_counts.max()\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the normalized data\nplt.plot(df['Day'], looting_news_df['Looting News'], label='Looting News', marker='o')\nplt.plot(df['Day'], state_of_emergency_news_df['State of Emergency News'], label='State of Emergency News', marker='o')\nplt.plot(df['Day'], protest_news_df['Protest News'], label='Protest News', marker='o')\nplt.plot(date_counts_normalized.index, date_counts_normalized.values, label='GDELT Articles (Normalized)', marker='o', color='blue')\n\n# Set y-axis range from 0 to 1 (normalized range)\nplt.ylim(0, 1.1)\n\n# Set x-axis ticks to match the dates\nplt.xticks(df['Day'], rotation=45)\n\n# Add vertical lines for specific dates\ndates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\ntitle_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\ncolors = [\"red\", \"green\"]\n\nfor date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n    ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n# Merge all legends into a single legend\nlegend = ax.legend(loc='upper left')\n\n# Add labels and title\nplt.xlabel('Date')\nplt.ylabel('Normalized Values')\nplt.title('Comparison of Normalized Data (News Search Trends and GDELT Articles)')\n\n# Show the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:59.342034Z","iopub.execute_input":"2023-11-08T11:17:59.342402Z","iopub.status.idle":"2023-11-08T11:17:59.890332Z","shell.execute_reply.started":"2023-11-08T11:17:59.342372Z","shell.execute_reply":"2023-11-08T11:17:59.88915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\nimport ipywidgets as widgets\nfrom ipywidgets import interact\n\n# Create a string with the CSV data for the news search trends\ncsv_data = \"\"\"Day,Looting_news,State of emergency_news,Protest_news\n2021-07-09,0,0,9\n2021-07-10,0,0,12\n2021-07-11,0,6,15\n2021-07-12,55,23,25\n2021-07-13,100,21,25\n2021-07-14,54,12,0\n2021-07-15,30,0,39\n2021-07-16,22,0,0\n2021-07-17,0,0,7\n2021-07-18,0,0,0\n2021-07-19,13,0,0\n2021-07-20,11,10,0\n2021-07-21,12,0,14\n\"\"\"\n\n# Create a DataFrame from the CSV data\ndf = pd.read_csv(StringIO(csv_data))\n\n# Convert \"Day\" column to datetime\ndf['Day'] = pd.to_datetime(df['Day'])\n\n# Create DataFrames for each curve\nlooting_news_df = pd.DataFrame({'Day': df['Day'], 'Looting News': df['Looting_news']})\nstate_of_emergency_news_df = pd.DataFrame({'Day': df['Day'], 'State of Emergency News': df['State of emergency_news']})\nprotest_news_df = pd.DataFrame({'Day': df['Day'], 'Protest News': df['Protest_news']})\n\n# Normalize each DataFrame to its own max value\nlooting_news_df['Looting News'] = looting_news_df['Looting News'] / looting_news_df['Looting News'].max()\nstate_of_emergency_news_df['State of Emergency News'] = state_of_emergency_news_df['State of Emergency News'] / state_of_emergency_news_df['State of Emergency News'].max()\nprotest_news_df['Protest News'] = protest_news_df['Protest News'] / protest_news_df['Protest News'].max()\n\n# Load the GDELT dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf_gdelt = pd.read_csv(file_path)\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df_gdelt['SQLDATE'].value_counts().sort_index()\n\n# Convert SQLDATE to the same datetime format as in the news search trends data\ndate_counts.index = pd.to_datetime(date_counts.index, format='%Y%m%d')\n\n# Normalize the GDELT data with its own max value\ndate_counts_normalized = date_counts / date_counts.max()\n\n# Create a function to update the plot based on the selected dataframes\ndef update_plot(show_looting, show_state, show_protest, show_gdelt):\n    fig, ax = plt.subplots(figsize=(12, 6))\n\n    if show_looting:\n        ax.plot(df['Day'], looting_news_df['Looting News'], marker='o', label='Looting News (Normalized)', color='orange')\n    if show_state:\n        ax.plot(df['Day'], state_of_emergency_news_df['State of Emergency News'], marker='o', label='State of Emergency News (Normalized)', color='green')\n    if show_protest:\n        ax.plot(df['Day'], protest_news_df['Protest News'], marker='o', label='Protest News (Normalized)')\n    if show_gdelt:\n        ax.plot(date_counts_normalized.index, date_counts_normalized.values, marker='o', label='GDELT Articles (Normalized)', color='blue')\n\n    # Set y-axis range from 0 to 1 (normalized range)\n    plt.ylim(0, 1.1)\n\n    # Set x-axis ticks to match the dates\n    plt.xticks(df['Day'], rotation=45)\n\n    # Add vertical lines for specific dates\n    dates_to_highlight = [\"2021-07-09\", \"2021-07-12\"]  # Add your desired dates here\n    title_of_dates = [\"Start of Unrest\", \"SANDF deployed\"]\n    colors = [\"red\", \"green\"]\n\n    for date_to_highlight, color, title in zip(dates_to_highlight, colors, title_of_dates):\n        ax.axvline(pd.to_datetime(date_to_highlight), color=color, linestyle='--', label=title)\n\n    # Merge all legends into a single legend\n    legend = ax.legend(loc='upper left')\n\n    # Add labels and title\n    plt.xlabel('Date')\n    plt.ylabel('Normalized Values')\n    plt.title('Comparison of Normalized Data (News Search Trends and GDELT Articles)')\n\n    plt.tight_layout()\n    plt.show()\n\n# Create checkboxes to select the dataframes\nshow_looting_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show Looting News',\n    disabled=False\n)\n\nshow_state_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show State of Emergency News',\n    disabled=False\n)\n\nshow_protest_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show Protest News',\n    disabled=False\n)\n\nshow_gdelt_checkbox = widgets.Checkbox(\n    value=False,\n    description='Show GDELT Articles',\n    disabled=False\n)\n\n# Create an interactive plot\ninteract(update_plot, show_looting=show_looting_checkbox, show_state=show_state_checkbox, show_protest=show_protest_checkbox, show_gdelt=show_gdelt_checkbox)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:17:59.892089Z","iopub.execute_input":"2023-11-08T11:17:59.89256Z","iopub.status.idle":"2023-11-08T11:18:00.383641Z","shell.execute_reply.started":"2023-11-08T11:17:59.892512Z","shell.execute_reply":"2023-11-08T11:18:00.38253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-five\"></a>\n# **5. Mathematical Framework**","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-five-a\"></a>\n## **Seasonal Decomposition**","metadata":{}},{"cell_type":"markdown","source":"## Components of decompsition:\n\n**1. Trend Component:**\n\nThe trend component represents the underlying **long-term behavior** or trend in the data. In this context, it represents whether data is showing an general trend of increasing, decreasing, or remaining relatively stable over time.\nBy examining the trend component, you can identify significant shifts or changes in the overall trend, such as periods of rapid growth or decline.\n\n**2. Seasonal Component:**\n\nThe seasonal component captures **repeating patterns** or seasonality in the data. Seasonality refers to regular fluctuations in the data that occur at fixed intervals, such as daily, weekly, monthly, or yearly patterns.\nBy analyzing the seasonal component, you can identify and understand the seasonal patterns in the number of GDELT articles. For example, you can determine if there are specific times of the year or week when article counts tend to be higher or lower.\n\n**3. Residual Component:**\n\nThe residual component represents the **unexplained or random variation in the data** that remains after removing the trend and seasonal components. It includes noise and irregular fluctuations.\nExamining the residual component can help you identify unusual or unexpected events that are not explained by the trend or seasonality. Large residual values may indicate outliers or anomalies in the data.","metadata":{}},{"cell_type":"markdown","source":"### Additive vs. Multiplicative Decomposition:\n\n- **Additive Decomposition:** Divides the data into three components - trend, seasonal, and residual. It's suitable when the seasonal effect remains constant over time.\n\n- **Multiplicative Decomposition:** Also separates data into trend, seasonal, and residual components, but in this method, they are multiplied together. Use this when the seasonal effect scales with the data's level.\n\n### \"Period\" Parameter:\n\nThe \"period\" parameter specifies the number of data points in one complete season. For daily data with annual seasonality, set \"period\" to 365.\n","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we utilized interactive widget functionality to test various combinations of seasonality frequency and decomposition method. This allowed us to dynamically visualize the decomposition results based on user-selected parameters.\n\nHowever, it's important to note that when viewing this notebook in Kaggle's static mode, interactive widgets are not functional. As a result, you won't be able to use the sliders and dropdowns to adjust the seasonality frequency and decomposition method in real-time. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\nfrom statsmodels.tsa.seasonal import seasonal_decompose\nimport ipywidgets as widgets\nfrom ipywidgets import interact\n\n# Load the GDELT dataset\nfile_path = \"/kaggle/input/south-africa-gdelt-riots-2021/Query_roits_2021_test6_allColumns.csv\"\ndf_gdelt = pd.read_csv(file_path)\n\n# Convert SQLDATE to the same datetime format as in the news search trends data\ndf_gdelt['SQLDATE'] = pd.to_datetime(df_gdelt['SQLDATE'], format='%Y%m%d')\n\n# Group the data by SQLDATE and count the number of data points for each date\ndate_counts = df_gdelt['SQLDATE'].value_counts().sort_index()\n\n# Cast the values in date_counts to integers\ndate_counts = date_counts.astype(int)\n\n# Create a function to decompose the time series with the chosen seasonality frequency and method\ndef decompose_and_plot(seasonality_frequency, decomposition_method):\n    # Decompose the time series with seasonality equal to 2 and the additive method\n    decomposition = seasonal_decompose(date_counts, model='additive', period=2)\n    \n    # Plot the original data\n    plt.figure(figsize=(12, 6))\n    plt.plot(date_counts.index, date_counts.values, marker='o', label='Number of GDELT Articles per Day', color='orange')\n    \n    # Plot the trend component\n    plt.plot(decomposition.trend.index, decomposition.trend.values, label='Trend', linestyle='--', color='blue')\n    \n    # Plot the seasonal component\n    plt.plot(decomposition.seasonal.index, decomposition.seasonal.values, label='Seasonal', linestyle='--', color='green')\n    \n    # Plot the residual component\n    plt.plot(decomposition.resid.index, decomposition.resid.values, label='Residual', linestyle='--', color='red')\n    \n    plt.xlabel('Date')\n    plt.ylabel('Count')\n    plt.title(f'Decomposition of GDELT Articles Time Series (Seasonality Frequency: 2, Method: Additive)')\n    plt.legend()\n    plt.show()\n\n# Create a slider widget for choosing seasonality frequency\nseasonality_frequency_slider = widgets.IntSlider(\n    value=3,  # Always set to 2 for this plot\n    min=1,\n    max=365,  # Adjust the maximum based on your data's seasonality\n    step=1,\n    description='Seasonality Frequency:',\n    continuous_update=False\n)\n\n# Create a dropdown widget for choosing decomposition method\ndecomposition_method_dropdown = widgets.Dropdown(\n    options=['Additive', 'Multiplicative'],\n    value='Additive',\n    description='Decomposition Method:',\n    disabled=False,\n)\n\n# Create an interactive plot\ninteract(decompose_and_plot, seasonality_frequency=seasonality_frequency_slider, decomposition_method=decomposition_method_dropdown)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:26:19.147681Z","iopub.execute_input":"2023-11-08T11:26:19.148095Z","iopub.status.idle":"2023-11-08T11:26:19.586963Z","shell.execute_reply.started":"2023-11-08T11:26:19.148065Z","shell.execute_reply":"2023-11-08T11:26:19.585802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-five-b\"></a>\n## **LOESS Regression:**\n\n####  LOESS — locally estimated scatterplot smoothing","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport statsmodels.api as sm\n\n# Resample the data by day and count the number of tweets\ntweets_per_day = unrest_tweets2.resample('D', on='date').size()\n\n# Normalize the first dataset (from (1)) to the range [0, 1]\nmax_value_tweets = tweets_per_day.max()\nnormalized_tweets = tweets_per_day / max_value_tweets\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(12, 6))\n\n# Plot the first dataset (Number of Tweets per Day)\nax.plot(tweets_per_day.index, normalized_tweets.values, marker='o', label='Number of Tweets per Day', color='skyblue')\n\n# Apply LOESS smoothing\nlowess_0_23999 = sm.nonparametric.lowess(normalized_tweets.values, tweets_per_day.index, frac=0.3499)  # You can adjust the 'frac' parameter to control the smoothing\nlowess_0_24 = sm.nonparametric.lowess(normalized_tweets.values, tweets_per_day.index, frac=0.4)  # You can adjust the 'frac' parameter to control the smoothing\n\n# Extract smoothed values\nsmoothed_values_0_23999 = lowess_0_23999[:, 1]\nsmoothed_values_0_24 = lowess_0_24[:, 1]\n\n# Plot the smoothed curve\nax.plot(tweets_per_day.index, smoothed_values_0_23999, label='LOESS Smoothed (frac=0.3499)', color='orange', linewidth=2)\nax.plot(tweets_per_day.index, smoothed_values_0_24, label='LOESS Smoothed (frac=0.4)', color='purple', linewidth=2)\n\n# Add labels and legend\nax.set_xlabel('Date')\nax.set_ylabel('Count')\nax.set_title('Number of Tweets per Day with LOESS Smoothing')\nax.legend()\n\nplt.savefig(\"SAUnrestLOESSReg.png\")\n\n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:01.454234Z","iopub.execute_input":"2023-11-08T11:18:01.45457Z","iopub.status.idle":"2023-11-08T11:18:03.692691Z","shell.execute_reply.started":"2023-11-08T11:18:01.454535Z","shell.execute_reply":"2023-11-08T11:18:03.691609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-five-c\"></a>\n## **Cross-Correlation between the News Media and Social Media** ","metadata":{}},{"cell_type":"markdown","source":"Cross-correlation is a technique used to measure the similarity between two signals or data sets as a function of the time lag between them.\n\n### What is Cross-Correlation?\n\n- Cross-correlation quantifies how well one signal matches another when one is shifted by a certain time lag.\n- It is used to identify patterns, periodicity, or relationships between two signals, even if they are not necessarily linearly related.\n- Cross-correlation can be used to find time delays, detect repeating patterns, or compare signals in various scientific fields.\n\n### Formula for Discrete Data\n\nThe formula for cross-correlation between two discrete data sets X and Y with n data points each is:\n\nR(k) = ∑(X(t)⋅Y(t−k))\n\n\nHere, `R(k)` represents the cross-correlation at time lag `k`, `X(t)` and `Y(t)` are discrete data points, and `∑` denotes summation over all data points.\n\n### Comparing Delay at Different Points\n\n- Cross-correlation allows us to compare the similarity or alignment between two signals at various time lags (`k`).\n- Positive peaks in the cross-correlation function indicate that one signal follows the other with a specific time delay (`k`).\n- Negative peaks suggest that one signal leads the other.\n- A time lag of `k = 0` means the signals are synchronized, while other values of `k` represent time offsets or delays.\n\nBy analyzing the cross-correlation function, researchers can determine the time lag at which the two signals are most similar or divergent, providing valuable insights into their temporal dynamics and relationships.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom scipy.signal import correlate\nfrom scipy.spatial.distance import euclidean\nimport matplotlib.pyplot as plt\n\n# Assuming you have loaded and normalized your datasets as described earlier\n# The two datasets are normalized_tweets and normalized_date_counts\n\n# Create a new DataFrame matching the length of normalized_tweets\nnormalized_date_counts_corr = pd.DataFrame({'Value': normalized_date_counts[:len(normalized_tweets)]})\n\n# Calculate Cross-Correlation\ncross_corr = correlate(normalized_tweets, normalized_date_counts_corr['Value'], mode='full', method='fft')\n\n# Calculate Time Lags\ntime_lags = np.arange(-len(normalized_tweets) + 1, len(normalized_date_counts_corr))\n\n# Find the lag at which the cross-correlation is the highest\nbest_lag = time_lags[np.argmax(cross_corr)]\n\n# Calculate Euclidean Distance and specify the best Euclidean distance at each shift\neuclidean_dists = []\nbest_euclidean_dist = float('inf')  # Initialize with a high value\n\nfor shift in range(len(normalized_tweets)):\n    shifted_date_counts = np.roll(normalized_date_counts_corr['Value'], shift)\n    euclidean_dist = euclidean(normalized_tweets, shifted_date_counts)\n    euclidean_dists.append(euclidean_dist)\n    \n    if euclidean_dist < best_euclidean_dist:\n        best_euclidean_dist = euclidean_dist\n\n# Plot the cross-correlation graph\nplt.figure(figsize=(12, 6))\nplt.plot(time_lags, cross_corr, marker='o', linestyle='-', color='blue')\nplt.xlabel('Lag (in days)')\nplt.ylabel('Cross-Correlation')\nplt.title('Cross-Correlation for Different Time Lags')\nplt.axvline(best_lag, color='red', linestyle='--', label=f'Best Lag: {best_lag}')\nplt.legend()\nplt.grid(True)\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:03.694112Z","iopub.execute_input":"2023-11-08T11:18:03.694502Z","iopub.status.idle":"2023-11-08T11:18:04.036377Z","shell.execute_reply.started":"2023-11-08T11:18:03.694452Z","shell.execute_reply":"2023-11-08T11:18:04.035203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Black Lives Matter (Kaggle):","metadata":{}},{"cell_type":"markdown","source":"Regrettably, this dataset proved to be a dead end as it lacked any pertinent information for our research. The data's irrelevance stemmed from its limited date range and misalignment with the timing of the original Black Lives Matter protests.","metadata":{}},{"cell_type":"code","source":"csv_file_path= '/kaggle/input/twitter-black-lives-matter-100k/100000_tweets_BlackLivesMatter_04-03-23.csv'\nblm_tweets = pd.read_csv(csv_file_path, low_memory=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:04.037665Z","iopub.execute_input":"2023-11-08T11:18:04.037973Z","iopub.status.idle":"2023-11-08T11:18:07.513159Z","shell.execute_reply.started":"2023-11-08T11:18:04.037947Z","shell.execute_reply":"2023-11-08T11:18:07.511957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"blm_tweets.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:07.51473Z","iopub.execute_input":"2023-11-08T11:18:07.515084Z","iopub.status.idle":"2023-11-08T11:18:07.53868Z","shell.execute_reply.started":"2023-11-08T11:18:07.515054Z","shell.execute_reply":"2023-11-08T11:18:07.53763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('There are {} rows and {} columns in the twitter dataset'.format(blm_tweets.shape[0],blm_tweets.shape[1]))","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:07.53985Z","iopub.execute_input":"2023-11-08T11:18:07.540408Z","iopub.status.idle":"2023-11-08T11:18:07.54622Z","shell.execute_reply.started":"2023-11-08T11:18:07.540378Z","shell.execute_reply":"2023-11-08T11:18:07.54519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"blm_tweets.sort_values(by='date').info()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:07.54756Z","iopub.execute_input":"2023-11-08T11:18:07.547919Z","iopub.status.idle":"2023-11-08T11:18:07.785711Z","shell.execute_reply.started":"2023-11-08T11:18:07.547891Z","shell.execute_reply":"2023-11-08T11:18:07.784416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Assuming you have a DataFrame called 'blm_tweets' with a 'date' column\n# First, make sure the 'date' column is in datetime format\nblm_tweets['date'] = pd.to_datetime(blm_tweets['date'])\n\n# Sort the DataFrame by the 'date' column\nblm_tweets_sorted = blm_tweets.sort_values(by='date')\n\n# Get the earliest date\nearliest_date = blm_tweets_sorted['date'].min()\n\n# Get the latest date\nlatest_date = blm_tweets_sorted['date'].max()\n\n# Format the dates in the desired format\nearliest_date_formatted = earliest_date.strftime('%Y-%m-%d')\nlatest_date_formatted = latest_date.strftime('%Y-%m-%d')\n\n# Print the formatted results\nprint(\"Earliest Date:\", earliest_date_formatted)\nprint(\"Latest Date:\", latest_date_formatted)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:07.787653Z","iopub.execute_input":"2023-11-08T11:18:07.788309Z","iopub.status.idle":"2023-11-08T11:18:08.393303Z","shell.execute_reply.started":"2023-11-08T11:18:07.788277Z","shell.execute_reply":"2023-11-08T11:18:08.392141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Assuming blm_tweets is your DataFrame with the 'date' column\nblm_tweets['date'] = pd.to_datetime(blm_tweets['date'])  # Convert the 'date' column to datetime\n\n# Resample the data by day and count the number of records (tweets)\nblm_tweets_per_day = blm_tweets.resample('D', on='date').size()\n\n# Find the date with the highest number of tweets\ndate_with_highest_tweets = blm_tweets_per_day.idxmax()\n\n# Create a figure and axis\nfig, ax = plt.subplots()\n\n# Plot the data\nax.plot(blm_tweets_per_day.index, blm_tweets_per_day.values, marker='o')\n\n# Set the x-axis format to display only the date in '%Y-%m-%d' format\ndate_formatter = DateFormatter('%Y-%m-%d')\nax.xaxis.set_major_formatter(date_formatter)\n\n# Set the date range dynamically based on the earliest and latest dates in the data\nearliest_date = blm_tweets['date'].min()\nlatest_date = blm_tweets['date'].max()\ndate_range = pd.date_range(start=earliest_date_formatted, end=latest_date_formatted, freq='D')\nall_dates = date_range.tolist()\n\nax.set_xticks(all_dates)\n\nax.legend()\n\n# Rotate the x-axis labels for better visibility\nplt.xticks(rotation=90)\n\n# Set labels and title\nax.set_xlabel('Date')\nax.set_ylabel('Number of Tweets')\nax.set_title('Number of Tweets per Day (US Election Data)')\n\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:08.395084Z","iopub.execute_input":"2023-11-08T11:18:08.39572Z","iopub.status.idle":"2023-11-08T11:18:08.804314Z","shell.execute_reply.started":"2023-11-08T11:18:08.395689Z","shell.execute_reply":"2023-11-08T11:18:08.803017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-six\"></a>\n# **6. Iran Protests 2022**\nThe Mahsa Amini Protests in Iran, which began on September 16, 2022, and persisted into 2023, were triggered by the death of Mahsa Amini. On September 13, 2022, the 22-year-old Mahsa Amini was arrested by Iran's morality police for \"improperly\" wearing her hijab. According to her family and local media reports, she was severely beaten during her arrest and subsequently died three days later while still in police custody. These protests stand out as a unique and powerful expression of dissent, challenging the Iranian government and constituting the most significant revolt the country has witnessed since the 1979 Islamic Revolution.\n","metadata":{}},{"cell_type":"code","source":"csv_file_path= '/kaggle/input/iran-protests-2022-tweets/tweets.csv'\niran_tweets = pd.read_csv(csv_file_path, low_memory=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:08.808147Z","iopub.execute_input":"2023-11-08T11:18:08.808497Z","iopub.status.idle":"2023-11-08T11:18:18.953933Z","shell.execute_reply.started":"2023-11-08T11:18:08.808454Z","shell.execute_reply":"2023-11-08T11:18:18.952557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iran_tweets.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:18.955277Z","iopub.execute_input":"2023-11-08T11:18:18.956031Z","iopub.status.idle":"2023-11-08T11:18:18.974784Z","shell.execute_reply.started":"2023-11-08T11:18:18.955993Z","shell.execute_reply":"2023-11-08T11:18:18.973535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('There are {} rows and {} columns in the twitter dataset'.format(iran_tweets.shape[0],iran_tweets.shape[1]))","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:18.976577Z","iopub.execute_input":"2023-11-08T11:18:18.97699Z","iopub.status.idle":"2023-11-08T11:18:19.004129Z","shell.execute_reply.started":"2023-11-08T11:18:18.97695Z","shell.execute_reply":"2023-11-08T11:18:19.002827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iran_tweets.sort_values(by='date').info()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:19.005297Z","iopub.execute_input":"2023-11-08T11:18:19.005687Z","iopub.status.idle":"2023-11-08T11:18:20.110327Z","shell.execute_reply.started":"2023-11-08T11:18:19.005655Z","shell.execute_reply":"2023-11-08T11:18:20.108731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iran_tweets.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:20.112223Z","iopub.execute_input":"2023-11-08T11:18:20.112977Z","iopub.status.idle":"2023-11-08T11:18:20.503104Z","shell.execute_reply.started":"2023-11-08T11:18:20.112931Z","shell.execute_reply":"2023-11-08T11:18:20.501924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Iran Profiling report**","metadata":{}},{"cell_type":"markdown","source":"Commented out because of the computational time taken to generate it","metadata":{}},{"cell_type":"code","source":"# import pandas as pd\n# from pandas_profiling import ProfileReport\n# import os\n# from IPython.display import display, HTML\n\n# # Define the report file name\n# report_filename = \"/kaggle/input/iran-profile-report/iran_report.html\"\n\n# # Check if the report file exists\n# if os.path.exists(report_filename):\n#     # Display the existing report by opening it in a web browser\n#     with open(report_filename, 'r', encoding='utf-8') as file:\n#         report_html = file.read()\n#         display(HTML(report_html))\n# else:\n#     # Generate the report and save it to the file\n#     profile = ProfileReport(iran_tweets, title='Iran Profiling Report', minimal=True)\n#     profile.to_file(report_filename)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:20.504709Z","iopub.execute_input":"2023-11-08T11:18:20.505035Z","iopub.status.idle":"2023-11-08T11:18:20.511278Z","shell.execute_reply.started":"2023-11-08T11:18:20.505008Z","shell.execute_reply":"2023-11-08T11:18:20.509917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Assuming you have a DataFrame called 'iran_tweets' with a 'parsed_date' column\n\n# Group the DataFrame by the 'parsed_date' column and count the rows for each date\ndate_counts = iran_tweets.groupby('date').size().reset_index(name='count')\n\n# Print the date counts\nprint(date_counts)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:20.512975Z","iopub.execute_input":"2023-11-08T11:18:20.513362Z","iopub.status.idle":"2023-11-08T11:18:21.233996Z","shell.execute_reply.started":"2023-11-08T11:18:20.51333Z","shell.execute_reply":"2023-11-08T11:18:21.232751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These strings are causing problems:\n![image.png](attachment:7f1dca41-79cb-4fb2-ab5c-22ec325b9369.png)","metadata":{},"attachments":{"7f1dca41-79cb-4fb2-ab5c-22ec325b9369.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAeEAAAAyCAYAAABrl2M7AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABZESURBVHhe7Z3NaxxJlsBf798gcLMu0EnQK3mwoA8FvmgQtTLYl8G+qWbw6OQ1q5FhBlWffFh8sgTNutBgdBKGKd/cp5HBmppedGmoBbOlxVYffBKUoM1qYU997X0vPjJfZGVmRWZGfUh+P5O4qlL5EREvXrz3IjPeF78gkMPPP/9sPgmCIAiCEJJ/MP8LgiAIgjBhZBAWBCHBBRz+cQmW/niIn2aYT4fweGkJHr/xv8v+i2J/H9OH/aXHcPjJfJ1hLt48hqUXffMtiWlbrLelrPKc7Kv9+yfmuzBWPAZh3mhmS+ucpkMkG18JROoxifOmCg0J/vCx0TnVto9/VR3qnPE5cQulgEigZ0iZ5XfQSaNlYLKdfTzKdLbqdcaYQh8I3x7LcH8X4OhdmVKQnBs9RXWRuK+k7hlvf5iDu99+gA8fOrBlfhH8yR7PyuPtCW+9ooYz27d3sSk5qNhWj2Bxs2G+a0i4Vs424HjX/Z3ov1iB1kInOmcHmgkLlQS3CbCZEBUU4pXtRejYe3kF0AxUIY3dY3M/x7ADLXgtlqDwWWIU9VA/nzGu3YXn2F+f35nMXc7d2YDF7deVjP6L81NozNfsN2WENlH7WT1I28ObZve0uPlwNu5jxsgbz6oQJBzdf9GE090ncH/e/GBYfoRC9WjZfONcwPnHBuzci/ct39qC7tnAfCOL46kapB/eMj8YlBDv3ke71HCzDltvBxAfGYoG1L40HxHHWk0M+q4la70s48WvtwHetmDF7veyzulYtJxNWCh5nBsJSITXjKXN70lZ1iZSsbLdBdhrDh1L53z85lDfs9oSEQYb6TBbbK2TIsEyn7D9HkaRLgMaYm8B2uv2vNxD1QrKXq+I5ZneHvb3JrShC61Vuz8up3uc65HQvv0T06b8uBH1mg8/HzvnKOiaWB99Jgfx9Ua0R6Id+X0m5SrXY0u2R+p5y/cBXd/mi8WUW52Z3Yvzd17t0YvrvYBckTdc32xDL+O+DlPuKZbzNjTxd7qv7vaKvu7Ja/x9CzqpOhJJ1Gl0r871WFuzsjRQF8f9h/erPHif8z3m8yF7PKuG9yAcK8qE0KPSb37cgSeFrNE5uL7QZaEdbPyX2FE/nkdC9pS83ZQCz11fhG63FwnbxZsDVKqncB5AYFTnUGVcgaPGE7h7Tf9OHelg3nrJH+C4cQRPbafGez3A8h+bfR8+PDfHLcND+v4KPfnbbL93I2KnfVkzx3Vga+8g6hRzd56ba+l9Q9Y5Kp/eLbMfr99+iZ3TeA3KituMLW/uRXS3WwAm4tFBZXMQKS5UpqsD2DDH0DVhnQ8YOKi9BHii9vlFEXQZ8G9v8yiLrTtSBtQGcZ13FlCJ+xgwme1hOhHVJRpYO9/b/Q8jg07vN9v3O3BK9Wb2Ee31HtTN/qh+POo1GyMjZjvePWV1PgIc1Jpolatj8V7BkYGM9iDlvXoEa1HZSXZWov7syBXJLWNUH3i82oJFFi3T5S/fB2rzDTg9p/Mbg5R+/GkA3YXryjvXbaXlx8FLzo+gpuqgeMRr+d6wXCiwPVrWo7V9Dn9WdYrt0zDl72waef/2LgA6FLBZ1/LHDO7IYDBlsWWgPhDdK17vCNujs4ltvY59k67BnJHudhMGD/Rxx7sAre88+o6NgKg+IkwKj0HYNozZSMDWrZWEHWQdoFMibLX8SCsALXhP0XTDZlcdDBXwM1IUsXJ0uPlQK2QjsE9hDQVmEa4bRVsFHo5e666YznABva6xXs01lTVrvfZr12FRWfmhLUccKL6x9UoWeBcGP6kvTofVnl3CCMEOf9+Gkii05Ns+qLRsCIqUYFTGkx5eQ1vyWdfcemCvoeWlWigLlQl6CBtMcVKkJDLS8qjQHo4niINKNxFh4REYGgSqh0G555GQq5EwD+paHdZue7QHDWKbG5FRouUKzGCXR34fuHh3hOeNZScEytim8386hxreI3mfbii3PI1da2CTM+BTfgbVderAzdrjy5ozICqMbjv/aL4nMSFgGqRj3EhJc4/fa9w/nMighbXH3Ndr0PDpO8JUKB6OpvCv+ai90Fg5xyEgHwXIvYDnUEelrjqYCtGwcCELZVmLnXssz79Gwbxdg+pdkzMH9QYbhBzPyWyRNW/L8QTgmb5nJ1IQHDJ82sx7nJDVyrwKvcUe5mxRsj1U9AXidibPwuwaF2rKBX0x6yGGnmsKS14fGAM0kOF/F++wD96rA/zQh8FZFxavVzV8qoJGzQN0RPB+RmOMLDLolF6Mp1/IwCdDI8+wpGm+Nut37gAtXBUKD8Jq4L29BnVUwG5YlIeACipo9OzoYStl2RmLMNpYKGvY0qYwaQsWI6s/FNry11a3Dp23nqWEoBy0x0F14FjWaVZxUSjEurcFdSo/egZorkTz1doQ8mdUx0+FyoBKJLxxkeWJoFF1m4fD9XRFo1Ev0M4Z7aHOzaIKFvISWUSl/x0qTv3RizL1SoOK9pCIPrwmI7YMZLiif0Z9MhfVjvG0hparBqx9PapW8/uA8rT4eZOU6QMU0fjYg9dnNSwXthl+7jG5H0UpOVfY6ETO/PzN+7DzMae8EbEM6igbGczoMaMuU1EU9TxLKw7rO2ivOfL8VVvpj0UhWYZCfScHE4Vz59gNefusV8/mrQWNxyCcCJl11+DYM7xpw3vKQzbebPRQBg+pqrnPjPDzEDxEc6DmdkKFwdw54eMo3Lj8SM8dRQ+W4BYNSE5oWNePMz9+7S5sbLJQrs+8poJFA9Q8nqkfdb5438rZ2vC8WB6kQFhZ0jtMApqbUtMQ5n5oC9SZaI4Nonq3ERRSXny6Qj9J7xX+HdUedG7yZKKyGGVL9YIDv22ng/mdYhGGEvVKZSfjRt8HyvJukSsymfKdElLtuMjkiuZx/Qzm3D6QPC9uTvnL9oG3FGcjI4UiU6fQ3rNGktVJ7oN9jpFYRs69ofvxnWfNg6I2XM51yFl7+1pOI520OoC1IpGSSK7009dx37H6kz+gyKYXM/cJueNZBWTZypmEOgMZGLMa8hWminoQih6U8zVci0MKh17HGGvI+VIjfVQIQ/E5YUEQrjh8OkZIh7xYGYCF6sggLAgCwqedCoT/BUGohISjBUEQBGFKiCcsCIIgCFNCBmFBEARBmBIyCHtA69SGfcWBoKcrr8Dj//Sk7qg1j9XfhK9DvX7wiGsLnx3j6a9jpHT/MPP48u7tpcZjEHbfE1ZbWqMbQUq+NxUtBTh0TOK8/LjEu560Oe8A2mupLeBAlrxugHfAZg3VHqXKdYmMBmrHK6CYeJKCyRgbyb4+xfZWfZFdf0yGXEisrgu/qI0wK2SPZ+Xx9oQnnsqQL/iOW7wgh75WtAj992twtFpdQanKXT91l+aTdyTDQIs5YH2GftpWL186vndlpwnJIyVGiZe0PA2WsjML6pM8aca0lyZtbC7CwCR5oeUrk/pltqDXuhahQ4vaeC1pySjdP/SKXDOfcvKKkDeeVSFIOHocqQyzUMs0skXo1eLxkJJerBD6vcit3NWDstOfRdaR2hIGgeO1Z1vy1uvxsqJzzpkMxUXfzTHx+t7598PR98ZX0aHNLec5qwNeBluu5O+KRDm8PXR+nNMW5K3jb3mp8xLXtPdE97n/xuzDv7f37efVJDxIdk/OeRP7sqElLIEl8EB1e2cDtt4eQY88QyoDnic9lSGSUUZdP/twmHpcTnIBA2/L1HqPtgBGMWUYmq9D7YwypmH/pOUrHf3iXnNYjrP6a9Lb5/daoRyfenC0UIdlWopyrxcdR7ohShFq2uwGfrb3m9c/0mTnf+y7LDxq5x3ZorKnRDfoXOYcThvj5if/nwfZ41k1vAfhiaYyJLgSZZ2I1tu1iwiQwJBlQgubF8qEMgRl7clflzYz/RmWX60sZL2HVwBN2ylIGaameHOhctjE3iOX4FTnzE5Hl4mxtpUVxxaF97G+tceZnQKQllBs2TpgadwIfWxKyjmE1rTldeMt4KYsal1xB5O8ga037p4XlWxOSsb2tkkJh0YKpe3zkytSbPlpF9soO7q9PFPnqfXBk5nBEmteY/9IT2WYX0bVVrT07NBx6FV9Ey8hmpQnGkwyUxnaerf7iqRkzIUG3gEa2D0Y4IC8bDMrqTrXS9bqa1LWs6fO4JLdX187EThXjsuXg5yBxVt0puGcw/pesP+YNvsP7IPW4cjrH0RSdr77bzMKmzX2i3llWu8qGTKGHJWOZ6fS92M2lI/UtI1CUDwGYRPysA2j1hC21hR2+OCpDBEniYMWPjeX7EBZs0ophLBMlNLLJyv9Wf8HNB6YZ6m8MGNMjE7xpj3LQuUonY5unIxI45YBpUtUxp23JV+RESkZ45RwbhrFfMiAy0+7WCl1XiYZqQxHpp1kKTIpSQLfZ40bNXDTsbaf60hRZjpP3H/4pxsZ+6oxd6cOg/UB1Hl7kNfJM62pNaTdpByZdW6SkZCX+19DKyTQ4G7PWaQcOnJhjXhqfyckHfVXHf37Av/5Elp2ojzNWFeLC2jgYPvy7FROVC8lnacQnuLh6HGnMhxCpxW0qDy3JmG19uJ0GK1SijOljFIy63jizJfT5m2UkGfp6cleQaIsXLd6utM7YcMxcRlSMiYHR8XoaE1E5TLqvukmg+dRELMZI4BSMn7zyzM2f119ziweGOheUub9E8+M0OaVyCUyNOrwn78iwyHWVaVTSyrDhxkFZIizkPQsobJLIf0f0LC5V4PBu76aGlRyhd7xpNN5CiUG4bGnMhxCp3jbUqEevCalTUMzQKX1Iyi8hJ5I9L0Uy3Af773thO380N5c+nEjU7wpUMlQ2CeKLozAIx1dZL3TVEEi/Vn5FG8ZKQBDoCIfFK4LZHVneePGCwpr8IRIu5iEohuo1FmWHp1blkdAGNQHbCrDgGVUqRYVOoyZlcpQp2T8R1PetJSM1sMs3r9SISPlrU4BmLvcXy7Y796/hw7Ws5VpXY7iqSUpGqZTFca6sJMISc8MKB9w9hp66PYsYz1Ct4f9xEx9+KTzpPljNDRSnyXJ24f1aefFi+ueq43HIJwI0UwilSHft9QEQE8zsnLJkqV5V7ufwuEBnpAlg0I9gRpdFzePMCkdRx2OHxcJobrXnBRvFvZ36QLMSJ4zkY7OSY+H9dpJWvOlU7xlpADMxcpOWso5V650O3u2o5UP/gAW79xYR6mp81Td8TIkjisFTdeUTLuYg0od+DGe5lBPSjtTFqx8fEqodBlRSd6IQ8q0qecUzDXzUhkqmfvzb83vaSkZdSgVhrz7EigDUhuuNH9NDznp6/oN8O5DlLqMVreUSy3Zh15KTmZlnL88hP8139PJ6x/5OLrV3rPvtM5eG05V1BGNvQWKZaIhSb9XTed5xckdzyoga0cLwmVDPZw33lSGwSHDKZDBLAhXieJzwoIgCN6YMKQMwIKQigzCgiCMEf2Q11VdVEUQqiLhaEEQBEGYEuIJC4IgCMKUkEFYEARBEKaEDMIM5/WFoUfPzesEAR5Jny5jKod6bcjzXWcPaClP/9enJgQ9lYyyUfy+TJ1nvCqk1usN/f5k4PYQLgPj69s3bogsjQuPQdi+y8a2NIVhFFRSAKKBbeiYxHn5cfY9ULa5786ZJy6Hfg+AXW3IeSeTsO86DuMueu65IIGtL76FVsSpZJejErRIhPkYkRwISg9iYbEyGVx2SkLvlAYnrT0Ucb+bOSNHqMgE+7YQDG9PeGZSGSpFTou3U0KB6UMKvVTKOVpUgY5RS8NtQYc+l1iDe9aZvXR0n3PKOTJeV2DwIDthgOABGpfTM+DQiHoT2NO9zJwclvbQ+y8CreBWkSDh6EmmMtSKcDrr/Q6XJz/lHA3Qj7HDxB6/f0iHDJj9k9jj5x62EzbHjRsv6rgRqfOy26UCKe0yKh1dXjk02ekj3egDr1deZ7QlOlpGyjmn3l7oVIa0GpNVtvx6SQWcVudFUs6p5V9DD9Cp/US/LuS1xrKQy48lEymQzNtVvkpFIkh+z8znFMbVt//9fXmd239BK7GNZ8Dr/3BkPpUhwApuAfAehPkyeI4SQiUzyVSGM4VHyjmbbEJ7yXxB/NG013tQJw8ZN1oa065P7K7ZTUsm2nR0msKp88ZKVjq60eXITEf36RAOWPTBXavcvpeqt+PdH510dHkp55KpDP+y+UWUtUaneMv2IMOmnBNmmi9r8E/mY1FI5t8b2UyLqMSG6Wx4aSFYfvQey1v1PXEzjRJwvrs2r5NZjBtlwOeMYR6D8CymMrxEsFSGKqFDgeQJcWo9PQhEndaZM0+mqqPjxpE6rzwqHV0zkY6OKFsOs4D/SmpkwX3WYGX7byzCkp9yrlwqQ82s1bkQCpKnMgNiLIdpoev0UCjKZ5Qjuga9Mp7yTIJ18SLHkUKjen9UWU9ew1GjAzsfJ5+digyjzOkHo8My95PDAB3oLOjoaBrFw9FTTmU4U1RNOVcKMnzabI5+NubG03DS0b1PWsJVymFl5wnAM1fRDaej+2e9g7hEKeeESZM1UJABdzqUEamWmjoVZdo4C/TMi4qAUYa0oWcPaDouJRTK85pfuwsP0wxBpXMuG6gT93IypP00gPaIqciL8xoaxstDD55V82ZriSimxciCMQ7IeYI0o5qeT6LkQ692svX9TwBr95Zz77PwIDztVIazRbGUc5QaDCqluEOwo/74RTzIa0MoFNZ6n0AoTCmcquXQURqSO+t5DqWja+F3w6VKOaeYYHtcRlBv3LiRNbdqng1IDQPafX+N96Gn1QIrN5xluPtgWIEO0pSyMvLo/301nWEjYItDzkVyOs6AA++T+QNtIKptUu0+bjnT6T4z+xk9fZ1nDONg93S7BQdv+nBO5zI/V2eQPieMsvC3edTTaBxYo+i0S8+1uPS/G8DiQgtWXoJOIzoEORotNT4MziBjwPcahG0D6W3qqQxtB1LhSztX7f/AU2hGppyLUqPptGlRSJmsKPp9lXJ2mmiCz9w3dtTf/+vfI29u5Wwt4JOuOpRKln+qcIaCQvJYjg0yYMqUw5EPLZP2mYRkOrrrz/6gfie5yUs593/mezq2D0w45Vzw9nDL0TXTQT7luPKoB/XivupsqJ/u84fZKA9v69fsbxbgdwv4/zqalaQL1ttR3S6tHqECfw5Por/VGw3SG2fmb77acqOJ9m2JjHnUWmrEsApl5IxkiQ3aVp8lyqM3/UR+7aX+/tX938BXfD9lBPu+Bgfme/MPX8X77P4PNH3ZhMGt5NiT5c2OZo6a0UbFaLP3je34SxfbxrYjXn+tcQQr//Jv8ND+LW7NvTa0KV+7GduGyqXe4tHjw8F8PLWYRNaOZpDCpFeqQj1dGPp8E4EGOMl4MztIewiToKicUaj2XT0KmdPDR2RYVM2jfVmg8pJT1YEewKNqfbP4nLBwRTERBlH4M4K0hzAJSsoZC9US9OCo8uwv6wO0BVERN/V/9QfoZBBOUjhkeFWwDzuJwp8NpD2mhxmYPgsdUFLOvqwBJF4pJIo/u8OnO0PMSdvpyrQt4LQlTaehL0yh6qG3Pgoi4WhBEAShMGq6Tc1ha+ihx88lHB0SGYQFQRAEYUpIOFoQBEEQpoQMwoIgCIIwJWQQFgRBEIQpIYOwIAiCIEwFgP8HAwdcNaBr9hkAAAAASUVORK5CYII="}}},{"cell_type":"code","source":"import pandas as pd\n\n# Assuming you have a DataFrame called 'iran_tweets' with a 'date' column\n\n# Convert the 'date' column to a string\niran_tweets['date'] = iran_tweets['date'].astype(str)\n\n# Filter the rows where the 'date' column starts with '2022' or '2023'\niran_tweets_filtered = iran_tweets[iran_tweets['date'].str.startswith(('2022', '2023'))]\n\n# Reset the index of the filtered DataFrame\niran_tweets_filtered.reset_index(drop=True, inplace=True)\n\n# Now, iran_tweets_filtered contains only the rows where the 'date' column starts with '2022' or '2023'\n# Group the DataFrame by the 'parsed_date' column and count the rows for each date\ndate_counts = iran_tweets_filtered.groupby('date').size().reset_index(name='count')\n\n# Print the date counts\nprint(date_counts)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:21.235932Z","iopub.execute_input":"2023-11-08T11:18:21.236432Z","iopub.status.idle":"2023-11-08T11:18:22.349179Z","shell.execute_reply.started":"2023-11-08T11:18:21.236385Z","shell.execute_reply":"2023-11-08T11:18:22.347893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Assuming you have a DataFrame called 'iran_tweets_filtered' with a 'date' column\n\n# Convert the 'date' column to datetime format\niran_tweets_filtered['date'] = pd.to_datetime(iran_tweets_filtered['date'])\n\n# Extract the date component from the 'date' column\niran_tweets_filtered['date'] = iran_tweets_filtered['date'].dt.date\n\n# Group the DataFrame by the 'date' column and count the dates for each unique date\ndate_counts = iran_tweets_filtered.groupby('date').size().reset_index(name='count')\n\n# Print the date counts\nprint(\"Count per date:\")\nprint(date_counts)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:22.350368Z","iopub.execute_input":"2023-11-08T11:18:22.350687Z","iopub.status.idle":"2023-11-08T11:18:25.956979Z","shell.execute_reply.started":"2023-11-08T11:18:22.350661Z","shell.execute_reply":"2023-11-08T11:18:25.955933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the earliest date\nearliest_date = date_counts['date'].min()\n\n# Get the latest date\nlatest_date = date_counts['date'].max()\n\n# Print the results\nprint(\"Earliest Date:\", earliest_date)\nprint(\"Latest Date:\", latest_date)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:25.96044Z","iopub.execute_input":"2023-11-08T11:18:25.960777Z","iopub.status.idle":"2023-11-08T11:18:25.966734Z","shell.execute_reply.started":"2023-11-08T11:18:25.96075Z","shell.execute_reply":"2023-11-08T11:18:25.965901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-six-a\"></a>\n## **Number of Tweets per Day**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Set the figure size\nplt.figure(figsize=(25, 6))\n\n# Plot the count per date\nplt.plot(date_counts['date'], date_counts['count'], marker='o', linestyle='-')\n\n# Set plot title and labels\nplt.title('Count of Tweets per Date - Iran Protest tweets')\nplt.xlabel('Date')\nplt.ylabel('Count')\n\n# Rotate x-axis labels for better readability (optional)\nplt.xticks(rotation=90)\n\n# Set the x-axis format to display only the date in '%Y-%m-%d' format\ndate_formatter = DateFormatter('%Y-%m-%d')\nax.xaxis.set_major_formatter(date_formatter)\n\ndate_range = pd.date_range(start=earliest_date, end=latest_date, freq='5D')\nall_dates = date_range.tolist()\n\nplt.xticks(all_dates)\n\n# Show the plot\nplt.grid(True)\nplt.tight_layout()\nplt.show()\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:25.968018Z","iopub.execute_input":"2023-11-08T11:18:25.968865Z","iopub.status.idle":"2023-11-08T11:18:26.787605Z","shell.execute_reply.started":"2023-11-08T11:18:25.968824Z","shell.execute_reply":"2023-11-08T11:18:26.786321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tweets_per_day)\nprint('There are {} rows in the twitter unrest dataset'.format(tweets_per_day.shape[0]))\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:26.789085Z","iopub.execute_input":"2023-11-08T11:18:26.789529Z","iopub.status.idle":"2023-11-08T11:18:26.797903Z","shell.execute_reply.started":"2023-11-08T11:18:26.789488Z","shell.execute_reply":"2023-11-08T11:18:26.796479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.dates import DateFormatter\n\n# Set the figure size\nplt.figure(figsize=(12, 6))\n\nn_days = 11\nstart_date = pd.to_datetime('2022-09-30')  # Convert the start_date to a datetime object\n\n# Reset the index without generating the 'level_0' column\ndate_counts.reset_index(drop=True, inplace=True)\n\n\n# Filter the DataFrame to include the specified number of days from the start date\nfiltered_data = date_counts[pd.to_datetime(date_counts['date']) >= pd.to_datetime(start_date)].head(n_days)\n\n# Plot the count per date for the first 10 days\nplt.plot(filtered_data['date'], filtered_data['count'], marker='o', linestyle='-')\n\n# Set plot title and labels\nplt.title(f'Count of Tweets per Date - Iran Protest tweets (First {n_days} Days)')\nplt.xlabel('Date')\nplt.ylabel('Count')\n\ndate_range = pd.date_range(start=start_date, periods=n_days, freq='D')\nall_dates = date_range.tolist()\n\nplt.xticks(all_dates)\n\n# Rotate x-axis labels for better readability (optional)\nplt.xticks(rotation=90)\n\n# Set the x-axis format to display only the date in '%Y-%m-%d' format\ndate_formatter = DateFormatter('%Y-%m-%d')\nax = plt.gca()\nax.xaxis.set_major_formatter(date_formatter)\n\n# Show the plot\nplt.grid(True)\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:26.799904Z","iopub.execute_input":"2023-11-08T11:18:26.800349Z","iopub.status.idle":"2023-11-08T11:18:27.208429Z","shell.execute_reply.started":"2023-11-08T11:18:26.800307Z","shell.execute_reply":"2023-11-08T11:18:27.20725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-six-b\"></a>\n## **Alignment of Signatures**\n\n### **Iran Protests vs SA Unrest**","metadata":{}},{"cell_type":"markdown","source":"### **Aligned by Higest Correlation**","metadata":{}},{"cell_type":"markdown","source":"### Create function to find most correlated parts of both series","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom scipy.signal import correlate\n\ndef find_best_matching_section(pattern_df_1, pattern_df_2):\n    \n    isdf_1biggerdf_2 = True\n    \n    if len(pattern_df_1) >= len(pattern_df_2):\n        short_pattern_df = pd.DataFrame({'Value': pattern_df_2})\n        long_pattern_df = pd.DataFrame({'Value': pattern_df_1})\n        isdf_1biggerdf_2 = True\n        \n    else:\n        short_pattern_df = pd.DataFrame({'Value': pattern_df_1})\n        long_pattern_df = pd.DataFrame({'Value': pattern_df_2})\n        isdf_1biggerdf_2 = False\n        \n    # Convert DataFrames to NumPy arrays\n    short_pattern = short_pattern_df['Value'].to_numpy()\n    long_pattern = long_pattern_df['Value'].to_numpy()\n\n    # Initialize variables to store best correlation and its position\n    best_correlation = -1  # Initialize with a low value\n    best_position = None\n\n    # Iterate through all possible alignments of the two patterns\n    for position in range(len(long_pattern) - len(short_pattern) + 1):\n        sub_long_pattern = long_pattern[position:position + len(short_pattern)]\n\n        # Resize short_pattern to match the length of sub_long_pattern\n        resized_short_pattern = short_pattern\n\n        correlation = np.corrcoef(sub_long_pattern, resized_short_pattern)[0, 1]\n\n        # Update the best correlation and its position if a better one is found\n        if correlation > best_correlation:\n            best_correlation = correlation\n            best_position = position\n\n    # Extract the best matching section from the longer pattern\n    best_matching_section = long_pattern[best_position:best_position + len(short_pattern)]\n    \n    cut_long_pattern = best_matching_section\n    \n\n    # Return the best correlation and best matching section\n    # Make sure the patterns are return in the original order they were inputted\n    # if longer pattern in first input then longer pattern is in first output pattern\n    \n    if isdf_1biggerdf_2:\n        return best_correlation, cut_long_pattern, short_pattern\n    else: \n        return best_correlation, short_pattern, cut_long_pattern\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:27.210016Z","iopub.execute_input":"2023-11-08T11:18:27.2108Z","iopub.status.idle":"2023-11-08T11:18:27.221545Z","shell.execute_reply.started":"2023-11-08T11:18:27.210755Z","shell.execute_reply":"2023-11-08T11:18:27.220365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Set Up data in correct formatting and form and find most correlated cuts from each pattern","metadata":{}},{"cell_type":"code","source":"# print(tweets_per_day)\niran_tweets_per_day = date_counts\nunrest_tweets_per_day_dict = {\"date\": [day for day in tweets_per_day.index],\n                             \"count\": tweets_per_day.values}\n# print(tweets_per_day.values)\n\n# Convert to the desired data structure (list of dictionaries)\n\n\n# Now, structured_array contains the data in a structured NumPy array\n# print(unrest_tweets_per_day_dict_array)\n# print(unrest_tweets_per_day_dict)\n# Create a DataFrame from the dictionary\nunrest_tweets_per_day = pd.DataFrame(unrest_tweets_per_day_dict)\n\n# unrest_tweets_per_day['count'] = unrest_tweets_per_day.values\n\n# Normalize the data\niran_tweets_per_day['Normalized_Count'] = (iran_tweets_per_day['count'] - iran_tweets_per_day['count'].min()) / (iran_tweets_per_day['count'].max() - iran_tweets_per_day['count'].min())\nunrest_tweets_per_day['Normalized_Count'] = (unrest_tweets_per_day['count'] - unrest_tweets_per_day['count'].min()) / (unrest_tweets_per_day['count'].max() - unrest_tweets_per_day['count'].min())\n\nbest_correlation, iran_tweets_per_day_filtered, unrest_tweets_per_day_filtered = find_best_matching_section(iran_tweets_per_day[\"Normalized_Count\"], unrest_tweets_per_day[\"Normalized_Count\"])\n\niran_tweets_per_day_filtered_normalized = (iran_tweets_per_day_filtered - iran_tweets_per_day_filtered.min()) / (iran_tweets_per_day_filtered.max() - iran_tweets_per_day_filtered.min())\nunrest_tweets_per_day_filtered_normalized = (unrest_tweets_per_day_filtered - unrest_tweets_per_day_filtered.min()) / (unrest_tweets_per_day_filtered.max() - unrest_tweets_per_day_filtered.min())\n\n# Sample data (replace with your actual data)\ndata = {\n    'Date': [i+1 for i in range(len(unrest_tweets_per_day_filtered))],\n    'Iran_Protests': iran_tweets_per_day_filtered_normalized,\n    'South_African_Unrest': unrest_tweets_per_day_filtered_normalized\n}","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:27.223488Z","iopub.execute_input":"2023-11-08T11:18:27.224023Z","iopub.status.idle":"2023-11-08T11:18:27.266993Z","shell.execute_reply.started":"2023-11-08T11:18:27.223979Z","shell.execute_reply":"2023-11-08T11:18:27.265875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom scipy.spatial.distance import euclidean, cityblock\nfrom scipy.stats import pearsonr\nfrom scipy.spatial import distance\nfrom scipy.spatial.distance import jaccard\n\n\n# Create a DataFrame from the data\ndf_iran_unrest = pd.DataFrame(data)\n\n# Normalize the signatures using Min-Max scaling\nscaler = MinMaxScaler()\ndf_iran_unrest[['Iran_Protests', 'South_African_Unrest']] = scaler.fit_transform(df_iran_unrest[['Iran_Protests', 'South_African_Unrest']])\n\n# Calculate cosine similarity between the normalized signatures\nsignature1 = df_iran_unrest['Iran_Protests'].values.reshape(1, -1)\nsignature2 = df_iran_unrest['South_African_Unrest'].values.reshape(1, -1)\n\ncosine_similarity_score = cosine_similarity(signature1, signature2)\nprint(f\"Cosine Similarity Score: {cosine_similarity_score[0][0]}\")\n\n# Calculate Pearson Correlation Coefficient\npearson_corr, _ = pearsonr(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\nprint(f\"Pearson Correlation Coefficient: {pearson_corr}\")\n\n# Calculate Euclidean Distance\neuclidean_distance = euclidean(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\nprint(f\"Euclidean Distance: {euclidean_distance}\")\n\n# Calculate Manhattan Distance (L1 Distance)\nmanhattan_distance = cityblock(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\nprint(f\"Manhattan Distance (L1 Distance): {manhattan_distance}\")\n\n# Calculate Jaccard Similarity (for binary or categorical data)\n# Example: Convert to binary using a threshold and calculate Jaccard similarity\nthreshold = 0.5\nbinary_iran = np.where(df_iran_unrest['Iran_Protests'] > threshold, 1, 0)\nbinary_unrest = np.where(df_iran_unrest['South_African_Unrest'] > threshold, 1, 0)\n\njaccard_similarity = 1 - distance.jaccard(binary_iran, binary_unrest)\nprint(f\"Jaccard Similarity: {jaccard_similarity}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:27.268602Z","iopub.execute_input":"2023-11-08T11:18:27.269013Z","iopub.status.idle":"2023-11-08T11:18:27.447982Z","shell.execute_reply.started":"2023-11-08T11:18:27.268974Z","shell.execute_reply":"2023-11-08T11:18:27.446659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(10, 6))\n\n# Set x-axis labels as 'Day 1', 'Day 2', ...\nx_labels = data['Date']\nplt.xticks(x_labels)\n\n# Plot the normalized data\nplt.plot(x_labels, data[\"Iran_Protests\"], label='Iran Protest', marker='o', linestyle='-')\nplt.plot(x_labels, data[\"South_African_Unrest\"], label='Unrest SA', marker='o', linestyle='-')\n\n# Set plot title and labels\nplt.title('Normalized Count of Tweets per Day (Iran protests vs SA Unrest) - Aligned by highest correlation')\nplt.xlabel('Day')\nplt.ylabel('Normalized Count')\n\n\n\n# Add a legend\nplt.legend()\n\n# Show the plot\nplt.grid(True)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:27.44973Z","iopub.execute_input":"2023-11-08T11:18:27.450225Z","iopub.status.idle":"2023-11-08T11:18:27.850587Z","shell.execute_reply.started":"2023-11-08T11:18:27.450196Z","shell.execute_reply":"2023-11-08T11:18:27.849672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Aligned by Peak**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n\n\n\n# Normalize the data\nfiltered_data['Normalized_Count'] = (filtered_data['count'] - filtered_data['count'].min()) / (filtered_data['count'].max() - filtered_data['count'].min())\nunrest_tweets_per_day_1['Normalized_Count'] = (unrest_tweets_per_day_1[\"count\"] - unrest_tweets_per_day_1[\"count\"].min()) / (unrest_tweets_per_day_1[\"count\"].max() - unrest_tweets_per_day_1[\"count\"].min())\n\n# Create a figure and axis\nfig, ax = plt.subplots(figsize=(10, 6))\n\n# Set x-axis labels as 'Day 1', 'Day 2', ...\nx_labels = ['Day {}'.format(i) for i in range(1, len(filtered_data) + 1)]\nplt.xticks(range(1, len(filtered_data) + 1), x_labels)\n\n# Plot the normalized data\nplt.plot(x_labels, filtered_data['Normalized_Count'], label='Iran Protest', marker='o', linestyle='-')\nplt.plot(x_labels, unrest_tweets_per_day_1['Normalized_Count'], label='Unrest SA', marker='o', linestyle='-')\n\n# Set plot title and labels\nplt.title('Normalized Count of Tweets per Day (Iran protests vs SA Unrest) Aligned by Peak')\nplt.xlabel('Day')\nplt.ylabel('Normalized Count')\n\n\n\n# Add a legend\nplt.legend()\n\n# Show the plot\nplt.grid(True)\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:27.851712Z","iopub.execute_input":"2023-11-08T11:18:27.852186Z","iopub.status.idle":"2023-11-08T11:18:28.248414Z","shell.execute_reply.started":"2023-11-08T11:18:27.852156Z","shell.execute_reply":"2023-11-08T11:18:28.24754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nlength_of_data = len(filtered_data['count'])\n\n# Sample data (replace with your actual data)\ndata = {\n    'Date': [i+1 for i in range(length_of_data)],\n    'Iran_Protests': filtered_data['Normalized_Count'],\n    'South_African_Unrest': unrest_tweets_per_day_1['Normalized_Count']\n}\n\n# Create a DataFrame from the data\ndf_iran_unrest = pd.DataFrame(data)\n\n# Convert the 'Date' column to a datetime object\ndf_iran_unrest['Date'] = pd.to_datetime(df_iran_unrest['Date'])\n\n# Set the 'Date' column as the index\ndf_iran_unrest.set_index('Date', inplace=True)\n\n# Calculate tweets per day for each event\ndf_iran_unrest['Iran_Tweets_Per_Day'] = df_iran_unrest['Iran_Protests']\ndf_iran_unrest['Unrest_Tweets_Per_Day'] = df_iran_unrest['South_African_Unrest']\n\n# Normalize the signatures using Min-Max scaling\nscaler = MinMaxScaler()\ndf_iran_unrest[['Iran_Tweets_Per_Day', 'Unrest_Tweets_Per_Day']] = scaler.fit_transform(df_iran_unrest[['Iran_Tweets_Per_Day', 'Unrest_Tweets_Per_Day']])\n\n# Calculate cosine similarity between the normalized signatures\nsignature1 = df_iran_unrest['Iran_Tweets_Per_Day'].values.reshape(1, -1)\nsignature2 = df_iran_unrest['Unrest_Tweets_Per_Day'].values.reshape(1, -1)\n\nsimilarity_score = cosine_similarity(signature1, signature2)\n\nprint(f\"Cosine Similarity Score: {similarity_score[0][0]}\")\n\n# You can now use the similarity score to quantify the similarity between the two events.\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.249518Z","iopub.execute_input":"2023-11-08T11:18:28.250446Z","iopub.status.idle":"2023-11-08T11:18:28.267832Z","shell.execute_reply.started":"2023-11-08T11:18:28.250414Z","shell.execute_reply":"2023-11-08T11:18:28.266524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-seven\"></a>\n# **7. Event Archetype**","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-seven-a\"></a>\n## **Twitter Signature Similarity Metrics**","metadata":{}},{"cell_type":"markdown","source":"### **Possible similarity metrics:**\n\n1. **Cosine Similarity**:\n\n   - **How it works**: Cosine similarity measures the cosine of the angle between two vectors in a multi-dimensional space. It quantifies the similarity in direction between two vectors, regardless of their magnitudes. It ranges from -1 (completely dissimilar) to 1 (identical), with 0 indicating orthogonal (uncorrelated) vectors.\n\n   - **When to use it**:\n     - Cosine similarity is often used when you want to compare the similarity between two vectors' directions or orientations.\n     - It is particularly useful when the magnitude or scale of the data is not important, and you want to focus on the patterns or trends in the data.\n     - Common applications include text document similarity, recommendation systems, and clustering based on feature vectors.\n\n2. **Pearson Correlation Coefficient**:\n\n   - **How it works**: The Pearson correlation coefficient measures the linear correlation between two variables. It assesses how well the two variables' relationship can be described by a straight line. It ranges from -1 (perfect negative linear correlation) to 1 (perfect positive linear correlation), with 0 indicating no linear correlation.\n\n   - **When to use it**:\n     - Pearson correlation is suitable when you want to quantify the strength and direction of a linear relationship between two continuous variables.\n     - It is commonly used in statistical analysis and hypothesis testing to assess associations between variables.\n     - It's useful when you want to understand how changes in one variable are associated with changes in another variable and whether this relationship is positive or negative.\n\n3. **Euclidean Distance**:\n\n   - **How it works**: Euclidean distance calculates the straight-line distance between two points in a multi-dimensional space. It considers both the magnitude and direction of the vectors. Smaller distances indicate greater similarity or proximity.\n\n   - **When to use it**:\n     - Euclidean distance is used when you want to measure the spatial or geometric distance between two data points.\n     - It is commonly applied in clustering algorithms (e.g., K-means) and dimensionality reduction techniques (e.g., Principal Component Analysis).\n     - It's suitable when you want to assess how close or far apart two points are in a multi-dimensional space, considering both the magnitude and direction..","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom scipy.spatial.distance import euclidean, cityblock\nfrom scipy.stats import pearsonr\nfrom scipy.spatial import distance\nfrom scipy.spatial.distance import jaccard\n\n# Sample data (replace with your actual data)\ndata = {\n    'Date': [i+1 for i in range(length_of_data)],\n    'Iran_Protests': filtered_data['Normalized_Count'],\n    'South_African_Unrest': unrest_tweets_per_day_1['Normalized_Count']\n}\n\n# Create a DataFrame from the data\ndf_iran_unrest = pd.DataFrame(data)\n\n# Normalize the signatures using Min-Max scaling\nscaler = MinMaxScaler()\ndf_iran_unrest[['Iran_Protests', 'South_African_Unrest']] = scaler.fit_transform(df_iran_unrest[['Iran_Protests', 'South_African_Unrest']])\n\n# Calculate cosine similarity between the normalized signatures\nsignature1 = df_iran_unrest['Iran_Protests'].values.reshape(1, -1)\nsignature2 = df_iran_unrest['South_African_Unrest'].values.reshape(1, -1)\n\ncosine_similarity_score = cosine_similarity(signature1, signature2)\nprint(f\"Cosine Similarity Score: {cosine_similarity_score[0][0]}\")\n\n# Calculate Pearson Correlation Coefficient\npearson_corr, _ = pearsonr(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\nprint(f\"Pearson Correlation Coefficient: {pearson_corr}\")\n\n# Calculate Euclidean Distance\neuclidean_distance = euclidean(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\nprint(f\"Euclidean Distance: {euclidean_distance}\")\n\n# # Calculate Manhattan Distance (L1 Distance)\n# manhattan_distance = cityblock(df_iran_unrest['Iran_Protests'], df_iran_unrest['South_African_Unrest'])\n# print(f\"Manhattan Distance (L1 Distance): {manhattan_distance}\")\n\n# # Calculate Jaccard Similarity (for binary or categorical data)\n# # Example: Convert to binary using a threshold and calculate Jaccard similarity\n# threshold = 0.5\n# binary_iran = np.where(df_iran_unrest['Iran_Protests'] > threshold, 1, 0)\n# binary_unrest = np.where(df_iran_unrest['South_African_Unrest'] > threshold, 1, 0)\n\n# jaccard_similarity = 1 - distance.jaccard(binary_iran, binary_unrest)\n# print(f\"Jaccard Similarity: {jaccard_similarity}\")\n\n# print(type(filtered_data['Normalized_Count']))","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.26921Z","iopub.execute_input":"2023-11-08T11:18:28.27005Z","iopub.status.idle":"2023-11-08T11:18:28.293105Z","shell.execute_reply.started":"2023-11-08T11:18:28.270017Z","shell.execute_reply":"2023-11-08T11:18:28.291712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. Cosine Similarity Score: 0.7782\n   - Cosine similarity measures the cosine of the angle between two vectors in a multi-dimensional space. In this case, it indicates that the two interest vs. time graphs are relatively similar, with a similarity score of approximately 0.7782. Since cosine similarity ranges from -1 (completely dissimilar) to 1 (identical), a score of 0.7782 suggests a relatively high degree of similarity between the two graphs.\n\n2. Pearson Correlation Coefficient: 0.4514\n   - The Pearson correlation coefficient measures the linear correlation between two variables. A value of 0.4514 indicates a positive correlation between the two interest vs. time graphs, but the correlation is moderate. It's not a strong linear relationship, but there is some positive association between the two sets of data.\n\n3. Euclidean Distance: 1.1552\n   - Euclidean distance calculates the straight-line distance between two points in a multi-dimensional space. In this context, a distance of 1.1552 indicates that the two graphs are relatively close in Euclidean space. Smaller distances suggest greater similarity, so this value suggests moderate similarity between the two graphs.\n\n4. Manhattan Distance (L1 Distance): 3.1493\n   - Manhattan distance (also known as L1 distance) calculates the sum of the absolute differences between corresponding elements of two vectors. A distance of 3.1493 indicates that the two graphs differ in their values across multiple data points. It is relatively higher than the Euclidean distance, suggesting that the graphs differ in their trends.\n\n5. Jaccard Similarity: 0.1429\n   - Jaccard similarity measures the similarity between two binary or categorical sets. A similarity of 0.1429 suggests that, when treating the data as binary (e.g., above/below a certain threshold), only about 14.29% of the data points match between the two graphs. This indicates relatively low similarity when considering a binary representation of the data.\n\nIn summary, these metrics collectively suggest that there is some degree of similarity between the two interest vs. time graphs, but the level of similarity varies depending on the metric used. Cosine similarity and Euclidean distance indicate relatively higher similarity, while the Pearson correlation coefficient suggests a moderate positive linear relationship. Manhattan distance and Jaccard similarity indicate relatively lower similarity. The choice of which metric to emphasize may depend on the specific context and goals of your analysis.","metadata":{}},{"cell_type":"code","source":"def convert_dates(date_str):\n    try:\n        date_obj = datetime.strptime(date_str, \"%a %b %d %H:%M:%S +0000 %Y\")\n        formatted_date = date_obj.strftime(\"%Y-%m-%d\")\n        return formatted_date\n    except ValueError:\n        print(date_str)\n        return None","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.294479Z","iopub.execute_input":"2023-11-08T11:18:28.294894Z","iopub.status.idle":"2023-11-08T11:18:28.308106Z","shell.execute_reply.started":"2023-11-08T11:18:28.294864Z","shell.execute_reply":"2023-11-08T11:18:28.306661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n\n\ndef align_and_plot_dataframes(dataf1, dataf2, date1, date2):\n    \n    if len(dataf1)<len(dataf2):\n        df1=dataf1\n        df2=dataf2\n        date1 = date1\n        date2 = date2\n    elif len(dataf1)>len(dataf2):\n        df1=dataf2\n        df2=dataf1\n        temp_date = date1\n        date1 = date2\n        date2 = temp_date\n    else:\n        df1_adjusted = {\"days\": np.arange(1, len(dataf1)+1),\n                   \"normalized_values\": dataf1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(1, len(dataf2)+1),\n                       \"normalized_values\": dataf2['Normalized_Count'].values}\n        # Plot the data\n        plt.figure(figsize=(10, 6))\n        plt.plot(df1_adjusted[\"days\"], df1_adjusted[\"normalized_values\"], label=\"df1\", marker='o', linestyle='-')\n        plt.plot(df1_adjusted[\"days\"], df2_adjusted[\"normalized_values\"], label=\"df2\", marker='o', linestyle='-')\n        plt.xlabel(\"Days\")\n        plt.ylabel(\"Normalized Count\")\n        plt.title(\"Normalized Count Comparison\")\n        plt.legend()\n        plt.show()\n        print(\"END\")\n        return\n        \n\n    # Find the index of the specified dates in each dataframe\n    idx1 = df1[df1['date'] == date1].index[0]\n    idx2 = df2[df2['date'] == date2].index[0]\n\n\n    # Calculate the differences in days between the end of each dataframe and the specified dates\n    diff1 = len(df1) - idx1\n    diff2 = len(df2) - idx2\n\n    \n    if idx2<idx1:\n        df1_adjusted = {\"days\": np.arange(1, idx1+diff1+1),\n                   \"normalized_values\": df1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(idx1-idx2+1, idx1+diff2+1),\n                       \"normalized_values\": df2['Normalized_Count'].values}\n        \n    elif idx2+diff1>idx2+diff2:\n        df1_adjusted = {\"days\": np.arange(idx2-idx1+1, idx2+diff1+1),\n                   \"normalized_values\": df1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(1, idx2+diff2+1),\n                       \"normalized_values\": df2['Normalized_Count'].values}\n        \n    elif idx2+diff2>idx2+diff1 and idx1<idx2:\n        df1_adjusted = {\"days\": np.arange(idx2-idx1+1, idx2+diff1+1),\n                   \"normalized_values\": df1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(1, idx2+diff2+1),\n                       \"normalized_values\": df2['Normalized_Count'].values}\n        \n    elif idx1 == idx2:\n        df1_adjusted = {\"days\": np.arange(1, len(df1)+1),\n                   \"normalized_values\": df1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(1, len(df2)+1),\n                       \"normalized_values\": df2['Normalized_Count'].values}\n        \n    else:\n        print(\"Something went wrong!\")\n\n\n    y1 = df1_adjusted[\"normalized_values\"]\n    y2 = df2_adjusted[\"normalized_values\"]\n    \n    x1 = df1_adjusted[\"days\"]\n    x2 = df2_adjusted[\"days\"]\n\n    # Plot the data\n    plt.figure(figsize=(10, 6))\n    plt.plot(x1, y1, label=\"df1\", marker='o', linestyle='-')\n    plt.plot(x2, y2, label=\"df2\", marker='o', linestyle='-')\n    plt.xlabel(\"Days\")\n    plt.ylabel(\"Normalized Count\")\n    plt.title(\"Normalized Count Comparison\")\n    plt.legend()\n    plt.show()\n\n# Example usage:\n\ndateSAUnrest = '2021-07-12'  # Change to your desired date for the first dataframe\ndateIran = '2022-10-03'  # Change to your desired date for the second dataframe\n\n# unrest_tweets_per_day['date'] = pd.to_datetime((unrest_tweets_per_day[\"date\"]).apply(convert_dates))\nunrest_tweets_per_day['date'] = pd.to_datetime(unrest_tweets_per_day[\"date\"]).dt.strftime('%Y-%m-%d')\niran_tweets_per_day['date'] = pd.to_datetime(iran_tweets_per_day['date']).dt.strftime('%Y-%m-%d')\n\nalign_and_plot_dataframes(unrest_tweets_per_day, iran_tweets_per_day, dateSAUnrest, dateIran)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.310195Z","iopub.execute_input":"2023-11-08T11:18:28.31074Z","iopub.status.idle":"2023-11-08T11:18:28.645045Z","shell.execute_reply.started":"2023-11-08T11:18:28.3107Z","shell.execute_reply":"2023-11-08T11:18:28.643862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n\n\n\ndef align_plot_cut_and_normalize_dataframes(dataf1, dataf2, date1, date2, title1, title2):\n    \n    isdataf1bigger = False\n    if len(dataf1)<len(dataf2):\n        df1=dataf1\n        df2=dataf2\n        isdataf1bigger = False\n    elif len(dataf1)>len(dataf2):\n        df1=dataf2\n        df2=dataf1\n        temp_date = date1\n        date1 = date2\n        date2 = temp_date\n        temp_title = title1\n        title1 = title2\n        title2 = temp_title\n        isdataf1bigger = True\n    else:\n        df1_adjusted = {\"days\": np.arange(1, len(dataf1)+1),\n                   \"normalized_values\": dataf1['Normalized_Count'].values}\n\n        df2_adjusted = {\"days\": np.arange(1, len(dataf2)+1),\n                       \"normalized_values\": dataf2['Normalized_Count'].values}\n        # Plot the data\n        plt.figure(figsize=(10, 6))\n        plt.plot(df1_adjusted[\"days\"], df1_adjusted[\"normalized_values\"], label=title1, marker='o', linestyle='-')\n        plt.plot(df1_adjusted[\"days\"], df2_adjusted[\"normalized_values\"], label=title2, marker='o', linestyle='-')\n        plt.xlabel(\"Days\")\n        plt.ylabel(\"Normalized Count\")\n        plt.title(f\"Normalized Count Comparison ({title1} vs {title2})\")\n        plt.legend()\n        plt.show()\n        \n        df1_returned = {\"days\": df1['date'],\n                   \"normalized_values\": df1['Normalized_Count']}\n        \n        df2_returned = {\"days\": df2['date'],\n                   \"normalized_values\": df2['Normalized_Count']}\n        \n        return df1_returned, df2_returned\n    \n    # Find the index of the specified dates in each dataframe\n    idx1 = df1[df1['date'] == date1].index[0]\n    idx2 = df2[df2['date'] == date2].index[0]\n    \n    # Calculate the differences in days between the end of each dataframe and the specified dates\n    diff1 = len(df1) - idx1\n    diff2 = len(df2) - idx2\n    \n    if idx1>idx2:\n        lower_bound_1 = idx1-idx2\n        higher_bound_1 = len(df1)\n        lower_bound_2 = 0\n        higher_bound_2 = idx2 + diff1\n    \n    elif idx2 + diff1 > idx2 + diff2:\n        lower_bound_1 = 0\n        higher_bound_1 = idx1 + diff2\n        lower_bound_2 = idx2 - idx1\n        higher_bound_2 = len(df2)\n    \n    elif idx2+diff2>idx2+diff1 and idx1<idx2:\n        lower_bound_1 = 0\n        higher_bound_1 = len(df1)\n        lower_bound_2 = idx2 - idx1\n        higher_bound_2 = idx2 + diff1\n        \n    elif idx1 == idx2:\n        lower_bound_1 = 0\n        higher_bound_1 = len(df1)\n        lower_bound_2 = 0\n        higher_bound_2 = len(df1)\n    \n    df1_normalized_values = (df1['Normalized_Count'][lower_bound_1:higher_bound_1] - df1['Normalized_Count'][lower_bound_1:higher_bound_1].min()) / (df1['Normalized_Count'][lower_bound_1:higher_bound_1].max() - df1['Normalized_Count'][lower_bound_1:higher_bound_1].min())\n    \n    df2_normalized_values = (df2['Normalized_Count'][lower_bound_2:higher_bound_2] - df2['Normalized_Count'][lower_bound_2:higher_bound_2].min()) / (df2['Normalized_Count'][lower_bound_2:higher_bound_2].max() - df2['Normalized_Count'][lower_bound_2:higher_bound_2].min())\n    \n    df1_adjusted = {\"days\": np.arange(1, len(df1_normalized_values)+1),\n                   \"normalized_values\": df1_normalized_values}\n    df1_returned = {\"days\": df1['date'][lower_bound_1:higher_bound_1],\n                   \"normalized_values\": df1_normalized_values}\n    \n    \n    df2_adjusted = {\"days\": np.arange(1, len(df1_normalized_values)+1),\n                    \"normalized_values\": df2_normalized_values}\n    df2_returned = {\"days\": df1['date'][lower_bound_2:higher_bound_2],\n                   \"normalized_values\": df2_normalized_values}\n    \n    \n    \n    # Determine the larger of the two differences\n    day_len = max(diff1, diff2)+ max(idx1,idx2)\n    \n    \n    # Create an array of numbers from 1 to day_len + 1\n    x = np.arange(1, day_len + 1)\n    \n    \n    y1 = df1_adjusted[\"normalized_values\"]\n    y2 = df2_adjusted[\"normalized_values\"]\n    \n    x1 = df1_adjusted[\"days\"]\n    x2 = df2_adjusted[\"days\"]\n    \n    # Plot the data\n    plt.figure(figsize=(10, 6))\n    plt.plot(x1, y1, label=title1, marker='o', linestyle='-')\n    plt.plot(x2, y2, label=title2, marker='o', linestyle='-')\n    plt.xlabel(\"Days\")\n    plt.ylabel(\"Normalized Count\")\n    plt.title(f\"Normalized Count Comparison ({title1} vs {title2})\")\n    plt.legend()\n    plt.show()\n    \n    if isdataf1bigger == False:\n        return df1_returned, df2_returned\n    else:\n        return df2_returned, df1_returned\n\n# Example usage:\n\ndateSAUnrest = '2021-07-12'  # Change to your desired date for the first dataframe\ndateIran = '2022-10-03'  # Change to your desired date for the second dataframe\n\nSAUnrest_cut, Iran_cut=align_plot_cut_and_normalize_dataframes(unrest_tweets_per_day, iran_tweets_per_day, dateSAUnrest, dateIran, \"SA Unrest\", \"Iran\")","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.646733Z","iopub.execute_input":"2023-11-08T11:18:28.647644Z","iopub.status.idle":"2023-11-08T11:18:28.962847Z","shell.execute_reply.started":"2023-11-08T11:18:28.647606Z","shell.execute_reply":"2023-11-08T11:18:28.961704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom scipy.spatial.distance import euclidean\nfrom scipy.stats import pearsonr\n\ndef calculate_similarity_metrics(data, key1, key2):\n    # Create a DataFrame from the data\n    data_df = pd.DataFrame(data)\n\n    # Normalize the signatures using Min-Max scaling\n    scaler = MinMaxScaler()\n    data_df[[key1, key2]] = scaler.fit_transform(data_df[[key1, key2]])\n\n    # Calculate cosine similarity between the normalized signatures\n    signature1 = data_df[key1].values.reshape(1, -1)\n    signature2 = data_df[key2].values.reshape(1, -1)\n\n    cosine_similarity_score = cosine_similarity(signature1, signature2)\n    print(f\"Cosine Similarity Score: {cosine_similarity_score[0][0]}\")\n\n    # Calculate Pearson Correlation Coefficient\n    pearson_corr, _ = pearsonr(data_df[key1], data_df[key2])\n    print(f\"Pearson Correlation Coefficient: {pearson_corr}\")\n\n    # Calculate Euclidean Distance\n    euclidean_distance = euclidean(data_df[key1], data_df[key2])\n    print(f\"Euclidean Distance: {euclidean_distance}\")","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.964515Z","iopub.execute_input":"2023-11-08T11:18:28.965169Z","iopub.status.idle":"2023-11-08T11:18:28.975347Z","shell.execute_reply.started":"2023-11-08T11:18:28.965128Z","shell.execute_reply":"2023-11-08T11:18:28.974334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_SAUnrest_Iran = {'SAUnrest': SAUnrest_cut[\"normalized_values\"].values,\n                     'Iran': Iran_cut[\"normalized_values\"].values}\n\nkey1 = 'SAUnrest'\nkey2 = 'Iran'\ncalculate_similarity_metrics(data_SAUnrest_Iran, key1, key2)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.976734Z","iopub.execute_input":"2023-11-08T11:18:28.977322Z","iopub.status.idle":"2023-11-08T11:18:28.99465Z","shell.execute_reply.started":"2023-11-08T11:18:28.977292Z","shell.execute_reply":"2023-11-08T11:18:28.993375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-seven-b\"></a>\n## **Archetype Curves Formation**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport statsmodels.api as sm\n\ndef plot_tweets_with_loess(data_dict, key1, key2, frac, showPatterns = False, showOriginalLOESS = False, Save = False):\n    # Create a figure and axis\n    fig, ax = plt.subplots(figsize=(12, 6))\n\n    # Plot the first dataset\n    if showPatterns:\n        ax.plot(np.arange(1, len(data_dict[key1]) + 1), data_dict[key1], marker='o', label=key1, color='skyblue')\n        ax.plot(np.arange(1, len(data_dict[key2]) + 1), data_dict[key2], marker='o', label=key2, color='skyblue')\n\n    # Apply LOESS smoothing to the first dataset\n    lowess_key1 = sm.nonparametric.lowess(data_dict[key1], np.arange(1, len(data_dict[key1]) + 1), frac=frac)\n\n    # Apply LOESS smoothing to the second dataset\n    lowess_key2 = sm.nonparametric.lowess(data_dict[key2], np.arange(1, len(data_dict[key2]) + 1), frac=frac)\n\n    # Extract smoothed values\n    smoothed_values_key1 = lowess_key1[:, 1]\n    smoothed_values_key2 = lowess_key2[:, 1]\n\n    # Plot the smoothed curves\n    ax.plot(np.arange(1, len(data_dict[key1]) + 1), ((smoothed_values_key1 + smoothed_values_key2) / 2) + 0.2, label=f'LOESS Smoothed Upper boundary', color='green', linewidth=2)\n    ax.plot(np.arange(1, len(data_dict[key2]) + 1), ((smoothed_values_key1 + smoothed_values_key2) / 2) - 0.2, label=f'LOESS Smoothed Lower boundary', color='green', linewidth=2)\n    ax.plot(np.arange(1, len(data_dict[key2]) + 1), (smoothed_values_key1 + smoothed_values_key2) / 2, label=f'LOESS Smoothed (Average)', color='black', linewidth=2)\n\n    \n    if showOriginalLOESS:                       \n        ax.plot(np.arange(1, len(data_dict[key1]) + 1), smoothed_values_key1, label=f'LOESS Smoothed ({key1})', color='purple', linewidth=2)\n        ax.plot(np.arange(1, len(data_dict[key2]) + 1), smoothed_values_key2, label=f'LOESS Smoothed ({key2})', color='orange', linewidth=2)\n\n    # Add labels and legend\n    ax.set_xlabel('Date', fontsize=14)\n    ax.set_ylabel('Count', fontsize=14)\n    ax.set_title(f'Number of Tweets per Day with LOESS Smoothing (Average of {key1} and {key2})', fontsize=18)\n    ax.legend(fontsize=14)\n    ax.grid()\n    \n    if Save:\n        name_of_save = f'Number of Tweets per Day with LOESS Smoothing (Average of {key1} and {key2}).png'\n        # Save the plot as a PNG file\n        plt.savefig(name_of_save)\n\n    # Show the plot\n    plt.show()\n\n# Example usage:\nkey1 = 'SAUnrest'\nkey2 = 'Iran'\nfrac = 0.4\nplot_tweets_with_loess(data_SAUnrest_Iran, key1, key2, frac, True, True)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:28.995929Z","iopub.execute_input":"2023-11-08T11:18:28.996575Z","iopub.status.idle":"2023-11-08T11:18:29.420819Z","shell.execute_reply.started":"2023-11-08T11:18:28.996541Z","shell.execute_reply":"2023-11-08T11:18:29.419673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_tweets_with_loess(data_SAUnrest_Iran, key1, key2, frac, Save = True)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:29.422152Z","iopub.execute_input":"2023-11-08T11:18:29.423003Z","iopub.status.idle":"2023-11-08T11:18:29.891537Z","shell.execute_reply.started":"2023-11-08T11:18:29.422961Z","shell.execute_reply":"2023-11-08T11:18:29.890299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport statsmodels.api as sm\n\ndef plot_tweets_with_loess(data_dict, key1, key2, frac, showPatterns = False, showOriginalLOESS = False, Save = False):\n    # Create a figure and axis\n    fig, ax = plt.subplots(figsize=(12, 6))\n\n    # Plot the first dataset\n    if showPatterns:\n        ax.plot(np.arange(1, len(data_dict[key1]) + 1), data_dict[key1], marker='o', label=key1, color='purple')\n        ax.plot(np.arange(1, len(data_dict[key2]) + 1), data_dict[key2], marker='o', label=key2, color='green')\n\n    # Apply LOESS smoothing to the first dataset\n    lowess_key1 = sm.nonparametric.lowess(data_dict[key1], np.arange(1, len(data_dict[key1]) + 1), frac=frac)\n\n    # Apply LOESS smoothing to the second dataset\n    lowess_key2 = sm.nonparametric.lowess(data_dict[key2], np.arange(1, len(data_dict[key2]) + 1), frac=frac)\n\n    # Extract smoothed values\n    smoothed_values_key1 = lowess_key1[:, 1]\n    smoothed_values_key2 = lowess_key2[:, 1]\n\n    # Plot the smoothed curves\n    ax.plot(np.arange(1, len(data_dict[key2]) + 1), (smoothed_values_key1 + smoothed_values_key2) / 2, label=f'Trend', color='black', linewidth=2)\n    ax.plot(np.arange(1, len(data_dict[key1]) + 1), ((smoothed_values_key1 + smoothed_values_key2) / 2) + 0.2, label=f'Upper boundary', color='blue', linewidth=2)\n    ax.plot(np.arange(1, len(data_dict[key2]) + 1), ((smoothed_values_key1 + smoothed_values_key2) / 2) - 0.2, label=f'Lower boundary', color='red', linewidth=2)\n   \n\n    data_dict[\"Archetype\"] = (smoothed_values_key1 + smoothed_values_key2) / 2\n    data_dict[\"Archetype-lower\"] = ((smoothed_values_key1 + smoothed_values_key2) / 2) - 0.2\n    data_dict[\"Archetype-higher\"] = ((smoothed_values_key1 + smoothed_values_key2) / 2) + 0.2\n\n    # Fill the area between the upper and lower bounds with gray color\n    ax.fill_between(np.arange(1, len(data_dict[key1]) + 1), ((smoothed_values_key1 + smoothed_values_key2) / 2) - 0.2, ((smoothed_values_key1 + smoothed_values_key2) / 2) + 0.2, color='gray', alpha=0.25)\n    \n    if showOriginalLOESS:                       \n        ax.plot(np.arange(1, len(data_dict[key1]) + 1), smoothed_values_key1, label=f'LOESS Smoothed ({key1})', color='purple', linewidth=2)\n        ax.plot(np.arange(1, len(data_dict[key2]) + 1), smoothed_values_key2, label=f'LOESS Smoothed ({key2})', color='orange', linewidth=2)\n\n        \n    # Set the Y-axis limits to go from 0 to 1\n    ax.set_ylim(0, 1)\n    ax.set_xlim(1, len(data_dict[key1]))\n    \n    \n    # Label every 2 days and the first and last days\n    x_ticks = np.arange(1, len(data_dict[key1]) + 1, 2)\n    x_labels = [str(x) for x in x_ticks]\n    x_labels[0] = \"1\"\n    print(len(x_ticks))\n    print(len(x_labels))\n    if x_labels[-1] != str(len(data_dict[key1]) + 1):\n        x_labels.append(str(len(data_dict[key1])))\n        # Append the value to the array\n        x_ticks = np.append(x_ticks, len(data_dict[key1]))\n        print(len(x_ticks))\n        print(len(x_labels))\n\n    ax.set_xticks(x_ticks)\n    ax.set_xticklabels(x_labels)\n    \n    # Add labels and legend\n    ax.set_xlabel('Days', fontsize=18)\n    ax.set_ylabel('Count', fontsize=18)\n    ax.set_title('Protest Social Media Archetype', fontsize=20)\n    ax.legend(fontsize=16)\n    ax.grid()\n    \n    if Save:\n        name_of_save = f'Number of Tweets per Day with LOESS Smoothing (Average of {key1} and {key2})4.png'\n        # Save the plot as a PNG file\n        plt.savefig(name_of_save)\n\n    # Show the plot\n    plt.show()\n\n# Example usage:\nkey1 = 'SAUnrest'\nkey2 = 'Iran'\nfrac = 0.4\nplot_tweets_with_loess(data_SAUnrest_Iran, key1, key2, frac, True, True)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:29.893375Z","iopub.execute_input":"2023-11-08T11:18:29.89416Z","iopub.status.idle":"2023-11-08T11:18:30.273124Z","shell.execute_reply.started":"2023-11-08T11:18:29.894123Z","shell.execute_reply":"2023-11-08T11:18:30.271874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_tweets_with_loess(data_SAUnrest_Iran, key1, key2, frac,showPatterns = True, Save = True)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:30.274856Z","iopub.execute_input":"2023-11-08T11:18:30.275276Z","iopub.status.idle":"2023-11-08T11:18:30.804184Z","shell.execute_reply.started":"2023-11-08T11:18:30.275237Z","shell.execute_reply":"2023-11-08T11:18:30.80302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_tweets_with_loess(data_SAUnrest_Iran, key1, key2, frac, Save = True)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:30.805704Z","iopub.execute_input":"2023-11-08T11:18:30.806674Z","iopub.status.idle":"2023-11-08T11:18:31.318992Z","shell.execute_reply.started":"2023-11-08T11:18:30.806633Z","shell.execute_reply":"2023-11-08T11:18:31.31773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data_SAUnrest_Iran)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.320344Z","iopub.execute_input":"2023-11-08T11:18:31.320775Z","iopub.status.idle":"2023-11-08T11:18:31.327511Z","shell.execute_reply.started":"2023-11-08T11:18:31.320743Z","shell.execute_reply":"2023-11-08T11:18:31.326371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-seven-c\"></a>\n## **Evaluation Metrics**","metadata":{}},{"cell_type":"code","source":"print(\"SA Unrest compared to archetype: \")\nkey1 = 'SAUnrest'\nkey2 = 'Archetype'\ncalculate_similarity_metrics(data_SAUnrest_Iran, key1, key2)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.329446Z","iopub.execute_input":"2023-11-08T11:18:31.32988Z","iopub.status.idle":"2023-11-08T11:18:31.348752Z","shell.execute_reply.started":"2023-11-08T11:18:31.329848Z","shell.execute_reply":"2023-11-08T11:18:31.347297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Iran compared to archetype: \")\nkey1 = 'Iran'\nkey2 = 'Archetype'\ncalculate_similarity_metrics(data_SAUnrest_Iran, key1, key2)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.350339Z","iopub.execute_input":"2023-11-08T11:18:31.350871Z","iopub.status.idle":"2023-11-08T11:18:31.370368Z","shell.execute_reply.started":"2023-11-08T11:18:31.350836Z","shell.execute_reply":"2023-11-08T11:18:31.369054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MSE**","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\ndef mse(y_true, y_pred):\n    # Calculate the Mean Squared Error (MSE)\n    mse = np.mean((y_true - y_pred) ** 2)\n\n    \n    return mse","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.371986Z","iopub.execute_input":"2023-11-08T11:18:31.37242Z","iopub.status.idle":"2023-11-08T11:18:31.387948Z","shell.execute_reply.started":"2023-11-08T11:18:31.372381Z","shell.execute_reply":"2023-11-08T11:18:31.386219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"key1 = 'SAUnrest'\nkey2 = 'Archetype'\nmse_value = mse(data_SAUnrest_Iran[key2], data_SAUnrest_Iran[key1])\nprint(\"Normalized Mean Squared Error (NMSE) - SA Unrest:\", mse_value)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.391765Z","iopub.execute_input":"2023-11-08T11:18:31.39225Z","iopub.status.idle":"2023-11-08T11:18:31.401437Z","shell.execute_reply.started":"2023-11-08T11:18:31.392205Z","shell.execute_reply":"2023-11-08T11:18:31.400165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"key1 = 'Iran'\nkey2 = 'Archetype'\nmse_value = mse(data_SAUnrest_Iran[key2], data_SAUnrest_Iran[key1])\nprint(\"Normalized Mean Squared Error (NMSE) - Iran:\", mse_value)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T11:18:31.403334Z","iopub.execute_input":"2023-11-08T11:18:31.40379Z","iopub.status.idle":"2023-11-08T11:18:31.411511Z","shell.execute_reply.started":"2023-11-08T11:18:31.403749Z","shell.execute_reply":"2023-11-08T11:18:31.410652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}