Masa_SB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "32a84f45",
   "metadata": {},
   "source": [
    "The theory of asymmetric information will always be an important topic in the financial sector. Nowadays, tech-savvy financial institutions are commonly using natural language processing (NLP), and especially named entity recognition (NER) to find patterns, behaviors, or signals for investment research.\n",
    "\n",
    "Japan is the third biggest economy by nominal GDP. It’s the land of conglomerates such as Toyota, Sony, Uniqlo, Softbank or Nintendo, and it’s the fifth biggest stock market. Japanese stocks have outperformed their European peers for years. Compared to sky-high valuations in the USA, the Japanese companies are high-quality firms and undervalued. And the Japanese market is probably the embodiment of asymmetric information due to the complexity of Japanese.\n",
    "\n",
    "## Chairman & CEO, SoftBank Group Corp, Masayoshi Son\n",
    "### ソフトバンクグループ（株）代表取締役 会長兼社長執行役員、孫 正義\n",
    "\n",
    "After a three-year hiatus, the most powerfull Japanese businessman and the biggest tech investor in the world, Masayoshi Son came back to Twitter to share his voice, but in Japanese...\n",
    "What did he say? Does it have any impact on Softbank, or his recent investments? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "146654c0",
   "metadata": {},
   "source": [
    "**On this project, we will try to understand:**\n",
    "- Specificities of natural language processing in Japanese \n",
    "- Informal speech of **Masayoshi Son (Softbank)** on Twitter for micro perspective **(Insights starting Chapter 7.)**\n",
    "- Formal language of **Governor Kuroda (Bank of Japan)** with PDF files for macro perspective\n",
    "\n",
    "**Technical goal:**\n",
    "Create a NLP pipeline in Japanese for NER based on a solid framework with a small, carefully designed dataset using tiny computing power\n",
    "\n",
    "**Challenge**: Not tweak split mode, fine tuning or hyper-parameters, as well as, avoid PyTorch, Keras and TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5558df7",
   "metadata": {},
   "source": [
    "# Natural Language Processing / 自然言語処理\n",
    "\n",
    "The natural language processing (NLP) can be break into three steps, text processing, feature extraction, and modeling.\n",
    "In simple words, NLP is to transform human language with various steps including cleaning, preprocessing, engineering a statistical model for particular tasks. I will not go into details of each steps but mostly explaining three key points for Japanese.\n",
    "\n",
    "## 1. Problem I / Tokenization トークン化\n",
    "### Splitting raw text into small units  /  生テキストを単語の集合に自動的に分割\n",
    "For the sake of an example, let’s try to tokenize one sentence in English and one tweet from Masayoshi Son with NLTK. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "81f013a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York', '.']"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import pandas as pd\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "english = 'Good muffins cost $3.88\\nin New York.'\n",
    "word_tokenize(english)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee384d3d",
   "metadata": {},
   "source": [
    "* English sentences can easily be cut into words (White space tokenization) with a common tool such as NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "5c0ed3d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['今日の決算発表の録画はこちらです。興味があるの方、ご覧ください。']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "japanese = '今日の決算発表の録画はこちらです。興味があるの方、ご覧ください。'\n",
    "word_tokenize(japanese)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d3ec90f",
   "metadata": {},
   "source": [
    "* However, as an agglutinative language, Japanese doesn’t have space to separate words. Basic morpholizers cannot cut sentences into linguistic units in Japanese (as well as Chinese, Korea, or Hindi)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d668ca16",
   "metadata": {},
   "source": [
    "## 2. Solution I / Japanese morphological analyzers  /  日本語を形態素解析\n",
    "Specific morphological analyzers based on dictionaries are mandatory to tokenize in Japanese. Written in C++, **MeCab** has been the most popular for years. Other morphological analyzers are available such as Kuromoji in Java, Suika in Ruby, JUMAN in Perl, Janome in Pure Python, and RakutenMa in JavaScript.\n",
    "\n",
    "### NLP framework, spaCy  /  自然言語処理フレームワーク、spaCy\n",
    "SpaCy is not a morphological analyzer, but a full NLP framework integrating pretrained pipelines in multiple languages including Japanese. Developed by Explosion, spaCy used to leverage UniDic and MeCab in the first version. Now, they are using SudachiDic and SudachiPy from Works Applications. We will use spaCy as a backbone for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "45015770",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy \n",
    "nlp = spacy.load('ja_core_news_md') \n",
    "doc = nlp('大西市長、了解致しました。')\n",
    "r_list = []\n",
    "for sent in doc.sents:\n",
    "    r_list = r_list + [[str(token.i), token.text, token.lemma_, token.pos_, token.tag_] for token in sent]\n",
    "df_list = pd.DataFrame(r_list, columns = ['token_no', 'text', 'lemma', 'pos', 'tag'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "35b102ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>token_no</th>\n",
       "      <th>text</th>\n",
       "      <th>lemma</th>\n",
       "      <th>pos</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>大西</td>\n",
       "      <td>大西</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>名詞-固有名詞-地名-一般</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>市</td>\n",
       "      <td>市</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>名詞-普通名詞-一般</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>長</td>\n",
       "      <td>長</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>接尾辞-名詞的-一般</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>、</td>\n",
       "      <td>、</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>補助記号-読点</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>了解</td>\n",
       "      <td>了解</td>\n",
       "      <td>VERB</td>\n",
       "      <td>名詞-普通名詞-サ変可能</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>致し</td>\n",
       "      <td>致す</td>\n",
       "      <td>AUX</td>\n",
       "      <td>動詞-非自立可能</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>まし</td>\n",
       "      <td>ます</td>\n",
       "      <td>AUX</td>\n",
       "      <td>助動詞</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>た</td>\n",
       "      <td>た</td>\n",
       "      <td>AUX</td>\n",
       "      <td>助動詞</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>。</td>\n",
       "      <td>。</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>補助記号-句点</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  token_no text lemma    pos            tag\n",
       "0        0   大西    大西  PROPN  名詞-固有名詞-地名-一般\n",
       "1        1    市     市   NOUN     名詞-普通名詞-一般\n",
       "2        2    長     長   NOUN     接尾辞-名詞的-一般\n",
       "3        3    、     、  PUNCT        補助記号-読点\n",
       "4        4   了解    了解   VERB   名詞-普通名詞-サ変可能\n",
       "5        5   致し    致す    AUX       動詞-非自立可能\n",
       "6        6   まし    ます    AUX            助動詞\n",
       "7        7    た     た    AUX            助動詞\n",
       "8        8    。     。  PUNCT        補助記号-句点"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "eab8fd3a",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"ja\" id=\"ba5e7efd56e34a798203ed05647d0a33-0\" class=\"displacy\" width=\"890\" height=\"317.0\" direction=\"ltr\" style=\"max-width: none; height: 317.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">大西</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"170\">市</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"170\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"290\">長、</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"290\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"410\">了解</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"410\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"530\">致し</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"530\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"650\">まし</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"650\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"770\">た。</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"770\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-0\" stroke-width=\"2px\" d=\"M62,182.0 62,142.0 287.0,142.0 287.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M62,184.0 L58,176.0 66,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-1\" stroke-width=\"2px\" d=\"M182,182.0 182,162.0 284.0,162.0 284.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M182,184.0 L178,176.0 186,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-2\" stroke-width=\"2px\" d=\"M302,182.0 302,162.0 404.0,162.0 404.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">obl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M302,184.0 L298,176.0 306,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-3\" stroke-width=\"2px\" d=\"M422,182.0 422,162.0 524.0,162.0 524.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M524.0,184.0 L528.0,176.0 520.0,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-4\" stroke-width=\"2px\" d=\"M422,182.0 422,142.0 647.0,142.0 647.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M647.0,184.0 L651.0,176.0 643.0,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-ba5e7efd56e34a798203ed05647d0a33-0-5\" stroke-width=\"2px\" d=\"M422,182.0 422,122.0 770.0,122.0 770.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-ba5e7efd56e34a798203ed05647d0a33-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M770.0,184.0 L774.0,176.0 766.0,176.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style='dep', jupyter=True, options={'compact':True,'distance':120})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd23ebb",
   "metadata": {},
   "source": [
    "* Using the tool displacy, we can visualize the sentence into linguistic units and syntactic dependencies. The name Onishi (大西) and the title \"City Mayor\" (市長) are well linked together. The SOV structure of Japanese language is also well understand with the verb \"comprehension\" (了解) in the polite form (致す) on the past tense (ました)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "f9e7956d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    大西\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       "\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    市長\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">TITLE_AFFIX</span>\n",
       "</mark>\n",
       "、了解致しました。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style='ent', options={'compact':True})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "010bd4dd",
   "metadata": {},
   "source": [
    "* On April 22 2020, Masayoshi Son responding to the mayor of Kumamoto. In this small sentence, we can see that the **pretrained Japanese NER** model perfectly recognized the Mayor of Kumamoto, Kazufumi Onishi (熊本市長 大西一史) and his title (市長). \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf34963",
   "metadata": {},
   "source": [
    "## 3. Problem II / Named Entity Recognition  / 固有表現抽出\n",
    "The Japanese writing system is based on three separate sets of characters: hiragana (平仮名 / ひらがな), katakana (片仮名 / カタカナ), and kanji (漢字). Modern Japanese is arguably the hardest language to process with a blend of Sino-Japanese, native Japanese, Latin script (romaji /ローマ字), and vocabulary from the westerners (Dutch, Portuguese, French, English, German). The numeral system is a mix of Arabic and traditional Chinese numerals. \n",
    "\n",
    "The Japanese digital world brought us emojis in Unicode : ) or Kaoji using Cyrillic alphabet. ∵･(ﾟДﾟ) and Greek alphabets ＿φ(°-°=). If you add regional slang, colloquialism, marketing contractions, and the constant evolution of the language, it becomes a beautiful nightmare for named entity recognition.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99196673",
   "metadata": {},
   "source": [
    "## 4. Solution II / Pretrained pipelines / 学習済みモデル\n",
    "**SpaCy (2.4)** is providing 3 powerful Japanese pipelines optimized for CPU in small (12MB), medium (41MB), and large (531MB). Based on Universal Dependencies, SudachiPy, and spaCy framework, **GINZA (4.0)** is also popular among data teams in Japan. Let's compare see how they compare on tricky sentences about Softbank. \n",
    "### 4.a spaCy Model / spaCyのモデル"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "75ef79ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクグループ\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "が外資系金融機関からの借り入れを増やしている。\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "は\n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    7月1日\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       "から、同社の5Gブランドである「\n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SoftBank 5G\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       "」と「\n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ドラえもん\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       "」がコラボレーションした新テレビCM。\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    福岡ソフトバンクホークス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "・\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    王\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">TITLE_AFFIX</span>\n",
       "</mark>\n",
       "\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    貞治\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       "\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    会長\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">TITLE_AFFIX</span>\n",
       "</mark>\n",
       "「君たちが歴史の中でも\n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    一番\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORDINAL</span>\n",
       "</mark>\n",
       "強い」と最大限の称賛。\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "(\n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9434)\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">QUANTITY</span>\n",
       "</mark>\n",
       "は、\n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクG\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       "(9984)の子会社、会長の\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    孫正義\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       "氏は創業者取締役です。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc1 = ('ソフトバンクグループが外資系金融機関からの借り入れを増やしている。'\n",
    "'ソフトバンクは7月1日から、同社の5Gブランドである「SoftBank 5G」と「ドラえもん」がコラボレーションした新テレビCM。'\n",
    "'福岡ソフトバンクホークス・王貞治会長「君たちが歴史の中でも一番強い」と最大限の称賛。'\n",
    "'ソフトバンク(9434)は、ソフトバンクG(9984)の子会社、会長の孫正義氏は創業者取締役です。')\n",
    "\n",
    "displacy.render(nlp(doc1),style='ent',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a16198d",
   "metadata": {},
   "source": [
    "Without modifying the split mode, the spaCy Japanese pipeline calibrated with the medium model is giving us incredible results. \n",
    "\n",
    "1. In the first sentence, **SoftBank Group (ソフトバンクグループ)** is recognized as a company. \n",
    "\n",
    "2. In the second sentence about tv spot (featuring Bruce Willis as Doraemon), the mobile company **Softbank (ソフトバンク)** is spotted perfectly with the **date (7月1日)**, the product **Softbank 5G** and collaboration with Japanese anime **Doraemon (ドラえもん)** as a product.\n",
    "\n",
    "3. In the third sentence, the NER model also have the baseball team **Fukuoka Softbank Hawks (福岡ソフトバンクホークス)**, with a small mistake on the chairman and baseball legend, **Sadaharu Oh (王貞治)** due to his name of Chinese origins. \n",
    "\n",
    "4. In the fourth sentence (from financial news), it's more tricky, the **Softbank (ソフトバンク)** with the ticker (9434) is good, but **Softbank Group (ソフトバンクG)** with the ticker (9984) is recognized as a product. And CEO of Softbank is spotted as a person **Masaoyshi Son 孫正義**. \n",
    "\n",
    "### 4.b Ginza Model / GiNZAのモデル"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "3f1b95f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクグループ\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Person</span>\n",
       "</mark>\n",
       "が外資系金融機関からの借り入れを増やしている。\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Pro_Sports_Organization</span>\n",
       "</mark>\n",
       "は\n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    7月1日\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Date</span>\n",
       "</mark>\n",
       "から、同社の\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    5Gブランド\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Product_Other</span>\n",
       "</mark>\n",
       "である「\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SoftBank 5G\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Product_Other</span>\n",
       "</mark>\n",
       "」と「\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ドラえもん\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Music</span>\n",
       "</mark>\n",
       "」がコラボレーションした新テレビCM。\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    福岡ソフトバンクホークス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Person</span>\n",
       "</mark>\n",
       "・\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    王貞治\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Person</span>\n",
       "</mark>\n",
       "\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    会長\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Position_Vocation</span>\n",
       "</mark>\n",
       "「君たちが歴史の中でも\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    一番\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Rank</span>\n",
       "</mark>\n",
       "強い」と最大限の称賛。\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Pro_Sports_Organization</span>\n",
       "</mark>\n",
       "(\n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9434\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Date</span>\n",
       "</mark>\n",
       ")は、\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクG\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GOE_Other</span>\n",
       "</mark>\n",
       "(\n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9984\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Date</span>\n",
       "</mark>\n",
       ")の子会社、\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    会長\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Position_Vocation</span>\n",
       "</mark>\n",
       "の\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    孫正義\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Person</span>\n",
       "</mark>\n",
       "\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    氏\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Title_Other</span>\n",
       "</mark>\n",
       "は創業者\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    取締役\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">Position_Vocation</span>\n",
       "</mark>\n",
       "です。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nlp_ginza = spacy.load('ja_ginza') \n",
    "displacy.render(nlp_ginza(doc1),style='ent',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9461de14",
   "metadata": {},
   "source": [
    "The **Ginza pretrained pipeline** is giving us interesting results. \n",
    "\n",
    "1. In the first sentence, **SoftBank Group (ソフトバンクグループ)** is recognized as a person. \n",
    "\n",
    "2. In the second sentence about the tv spot, the mobile company **Softbank (ソフトバンク)** is spotted as a sport team and Japanese anime character **Doraemon (ドラえもん)** as a music. \n",
    "\n",
    "3. In the third sentence, the pipeline gave us the baseball team **Fukuoka Softbank Hawks (福岡ソフトバンクホークス)** as a person. The former coach, chairman and baseball legend, **Sadaharu Oh (王貞治)** is perfectly spotted as well as his title. \n",
    "\n",
    "4. In the fourth sentence, the mobile company **Softbank (ソフトバンク)** is recognized as a sports team. The **Softbank Group (ソフトバンクG)** as Geographical Entity (GEO). CEO of Softbank is spotted as a person **Masaoyshi Son 孫正義**. The honorific label for name **Sir(氏)**, the company **company director (取締役)** are both recognized. However, the title **founder (創業者)** is missing in the middle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9e5c366",
   "metadata": {},
   "source": [
    "## 5. Problem III / Wrong NER labels  / 間違えてしまった固有表現タグ\n",
    "While the Japanese pipeline by spaCy give us great results but doesn't separate entities. The Ginza pipeline give us separate entities, unfortunately the wrong ones.\n",
    "\n",
    "Depending on who you ask **“What is Softbank?”**, you will get a different answer. The Japanese people will tell you it’s a telecom company. A baseball fan will tell you it’s the best team in Japan since five years. For people working in the finance sector, Softbank is two giant tech funds, Vision Fund I & II. \n",
    "\n",
    "Based on media outlets and news, pretrained models are reproducing errors of humans. The Japanese financial press is writing **SoftbankG (ソフトバンクG)** for Softbank Group. Even Masayoshi Son on Twitter is using the contraction **SB**. Apparently, the Ginza model was trained on the data of website **Livedoor** by the data team at **Recruit Holdings** (Indeed/Glassdoor) which is perfect for titles and job position. However, a model trained on lot of articles on baseball using the word **Softbank (ソフトバンク)** to describe the Japanese baseball team, **Fukuoka Softbank Hawks (福岡ソフトバンクホークス)** will give us the wrong entities."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae0bdd2",
   "metadata": {},
   "source": [
    "## 6. Solution III / Industry specific labeling  / 依存関係のラベリング\n",
    "\n",
    "Regardless of the amazing capabilities of spaCy and Ginza, NER needs to be adjusted to be domain specific. We can have three approaches, the traditional rules-based model, bespoke statistical model with prior knowledge, or a neural approach but costly in terms of computing power.\n",
    "\n",
    "I have trained a NER bespoke model with a small training dataset in order to recognized particular entities:\n",
    "\n",
    "1. **Softbank** (company) 2. **Softbank Hawks** (baseball team) 3. **Softbank Vision Fund** \n",
    "4. **Masayoshi Son** 5. **Softbank tickers** 6. **Sofbank 5G products** \n",
    "7. **Sars-Cov2**  8. **Vaccin** \n",
    "\n",
    "### 6.a Bespoke Model / ビスポークのモデル "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "e2eff66a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクグループ\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "が外資系金融機関からの借り入れを増やしている。\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">BASEBALL</span>\n",
       "</mark>\n",
       "は7月1日から、同社の\n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    5G\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       "ブランドである「\n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SoftBank 5G\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       "」と「ドラえもん」がコラボレーションした新テレビCM。\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    福岡ソフトバンクホークス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">BASEBALL</span>\n",
       "</mark>\n",
       "・王貞治会長「君たちが歴史の中でも一番強い」と最大限の称賛。\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンク\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "(\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9434\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">TICKER</span>\n",
       "</mark>\n",
       ")は、\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ソフトバンクG\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "(\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9984\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">TICKER</span>\n",
       "</mark>\n",
       ")の子会社、会長の\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    孫正義\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       "氏は創業者取締役です。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nlp_bespoke = spacy.load(r\"SBModel\") #Loading bespoke model\n",
    "displacy.render(nlp_bespoke(doc1),style='ent',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bba4558",
   "metadata": {},
   "source": [
    "* My simple bespoke NER model in Japanese is able to recognized the business entities companies, tickers and products, as well as separate the baseball team. The model is still a little bit confused on the Softbank as a baseball team on the second sentence. \n",
    "***\n",
    "***- Now let's see try the model on few tweets of Masayoshi Son !***\n",
    "\n",
    "\n",
    "## 7. Tweets of Masayoshi Son / 孫正義氏のツイート\n",
    "\n",
    "With an impressive 2.8M followers, the last two tweets from Masayoshi Son were from October 2015 about the Hawks’ victory and in February 2017 for the announcement of an environmental grant.\n",
    "\n",
    "The Japanese investor came back strong with 110 tweets from March 10th, 2020 to June 6th 2021. In the first three tweets, we can easily understand that Masayoshi Son was concerned about the Sars-Cov2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "c45722f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserScreenName</th>\n",
       "      <th>UserName</th>\n",
       "      <th>Date</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>孫正義</td>\n",
       "      <td>@masason</td>\n",
       "      <td>2020-03-10 12:15:50</td>\n",
       "      <td>久しぶりのツイートです。新型コロナウイルスの状況を心配しています。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>孫正義</td>\n",
       "      <td>@masason</td>\n",
       "      <td>2020-03-11 02:52:50</td>\n",
       "      <td>行動を開始します。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>孫正義</td>\n",
       "      <td>@masason</td>\n",
       "      <td>2020-03-11 09:46:03</td>\n",
       "      <td>本日厚労省を訪問しました。医療崩壊を起こさないよう連携しながらやっていきたい。#コロナ検査有志</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  UserScreenName  UserName                 Date  \\\n",
       "0            孫正義  @masason  2020-03-10 12:15:50   \n",
       "1            孫正義  @masason  2020-03-11 02:52:50   \n",
       "2            孫正義  @masason  2020-03-11 09:46:03   \n",
       "\n",
       "                                              Text  \n",
       "0                久しぶりのツイートです。新型コロナウイルスの状況を心配しています。  \n",
       "1                                        行動を開始します。  \n",
       "2  本日厚労省を訪問しました。医療崩壊を起こさないよう連携しながらやっていきたい。#コロナ検査有志  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import csv\n",
    "df_masa = pd.read_csv(\"MasaSB.csv\", sep=\";\",encoding='utf-8')\n",
    "df_masa.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4bd6a9",
   "metadata": {},
   "source": [
    "## 8. Haiku / 一句\n",
    "\n",
    "Often pictured by the western media as \"*not your typical Japanese business man*\". Masayoshi Son’s Twitter feed is telling us another story. Masayoshi Son is using the polite form (丁寧語) to speak with his followers. While speaking to government officials, he is using the humble form (謙譲語) by respect. We can see that the Japanese CEO is supporting his local baseball team, takes pictures of cherry blossoms, and write haiku."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "6594a48d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "十六で\n",
      "志立て単身渡米。\n",
      "今の正直な心境を一句\n",
      "したためてみました。\n",
      "\n",
      "飛び跳ねて\n",
      "田で鳴くカエル\n",
      "空遠し。\n"
     ]
    }
   ],
   "source": [
    "print(df_masa['Text'][95])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e8bf7b1",
   "metadata": {},
   "source": [
    ">\"At sixteen, I had the will to go alone in USA. I will try to write a haiku about my present state of mind:\n",
    "\n",
    ">**Jump up and down**\n",
    "\n",
    ">**Croaking frogs in the rice field**\n",
    "\n",
    ">**Far away\"**\n",
    "\n",
    "  *Masayoshi Son*\n",
    "\n",
    "*Note: the standard NLP preprocessing (delete the `\\n`) would have destroy the structure of this poetic tweet. : )*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "123423d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = df_masa['Text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "854e96bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re  #Tweets Cleaning \n",
    "def clean_text(text):\n",
    "    text = re.sub('\\n', '', text)\n",
    "    text = re.sub('\\u3000', '', text)\n",
    "    text = re.sub(\"@[A-Za-z0-9]+\",\"\",text)\n",
    "    text = re.sub(\"#\",\"\",text)\n",
    "    text = re.sub(\"(?:\\@|http?\\://|https?\\://|www)\\S+\", \"\", text)\n",
    "    return text\n",
    "df_masa['Text'] = df_masa['Text'].map(lambda x: clean_text(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "8608756f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                    久しぶりのツイートです。新型コロナウイルスの状況を心配しています。\n",
       "1                                            行動を開始します。\n",
       "2       本日厚労省を訪問しました。医療崩壊を起こさないよう連携しながらやっていきたい。コロナ検査有志\n",
       "3                             現在検討中の簡易PCR検査の流れ コロナ検査有志\n",
       "4    検査したくても検査してもらえない人が多数いると聞いて発案したけど、評判悪いから、やめようかな...\n",
       "5    新型コロナウイルスに不安のある方々に、簡易PCR検査の機会を無償で提供したい。まずは100万...\n",
       "Name: Text, dtype: object"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e951c5e7",
   "metadata": {},
   "source": [
    "### Test of custom NER model with 4 tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "d2f4f40e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">人種差別は悲しい事。黒人やラテン系起業家の成功を妨げる不公平な世の中を打開すべく、\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SBグループ\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "は１億ドル(100億円)のファンド (Opportunity Growth Fund)を立ち上げます。BlackLivesMatter</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "blm_fund = df_masa['Text'][62]\n",
    "doc4 = nlp_bespoke(blm_fund)\n",
    "displacy.render(doc4,style=\"ent\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b601eb24",
   "metadata": {},
   "source": [
    "* The custom model is spotting **Softbank Group (SBグループ)** as a company, but not the new fund."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "661e65a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">福岡PayPayドームでの\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ホークス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">BASEBALL</span>\n",
       "</mark>\n",
       "の試合は緊急事態宣言解除の日まで自主的に無観客試合とすることを検討したいと思います。正式発表は後ほど。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paypay = df_masa['Text'][103]\n",
    "doc5 = nlp_bespoke(paypay)\n",
    "displacy.render(doc5,style=\"ent\",jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "283810c5",
   "metadata": {},
   "source": [
    "* The NER model is spotting **Hawks (ホークス)** as the baseball team without being confused of the **Fukuoka PayPay Stadium (福岡PayPayドーム)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "ceef5f71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">マジか⁈ 「ハゲていると\n",
       "<mark class=\"entity\" style=\"background: #ffeb80; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    新型コロナウイルス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">EVENT</span>\n",
       "</mark>\n",
       "感染症が重症化する」という可能性発表- GIGAZINE</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hage = df_masa['Text'][75]\n",
    "doc6 = nlp_bespoke(hage)\n",
    "displacy.render(doc6,style=\"ent\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9880a7d",
   "metadata": {},
   "source": [
    "* The entity **Sars Cov-2 (新型コロナウイルス)** was perfectly found as an event inside a confusing joke linking covid-19 and baldness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "7f884253",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">医療従事者不足の中、大規模会場は\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    ワクチン\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">VACCIN</span>\n",
       "</mark>\n",
       "の作業効率向上に役立つのかも。だとしたら福岡ドームを無償提供検討したいけど如何でしょうか？</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vaccin = df_masa['Text'][102]\n",
    "doc7 = nlp_bespoke(vaccin)\n",
    "displacy.render(doc7,style=\"ent\",jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae971bf6",
   "metadata": {},
   "source": [
    "* And last sentence, the entity **vaccin (ワクチン)** was also found correctly. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d7fd3a",
   "metadata": {},
   "source": [
    "## 9. Insights / 洞察\n",
    "\n",
    "* Covid-19:\n",
    "\n",
    "In 110 tweets, Masayoshi Son mostly spoke about Covid-19 with 77 tweets. He connected with government officials to provide assistance and even the Fukuoka PayPay stadium. The Japanese CEO used his companies to obtain masks, test kits, and other necessities.\n",
    "\n",
    "* Business:\n",
    "\n",
    "Masayoshi Son only talked 4 times about Softbank Group as a company with 2 tweets for the announcement of financial results, and 1 tweet to respond about the first deficit of Softbank. 1 tweet to announce that the SoftBank group will launch a `$100` million fund **\"Opportunity Growth Fund\"** for minority-owned startups in response to the Black Lives Matter movement. \n",
    "\n",
    "* Other facts:\n",
    "\n",
    "6 tweets to celebrate the championship title of Softbank Hawks. 3 tweets to share his concerns about the upcoming Tokyo Olympics. Only 1 haiku, but more than 10 tweets with inspiring quotes. He shared one picture of cherry blossom, one of lotus flower, one of his brother and one joke about baldness. Since Masayoshi Son is not using Twitter as a digital tool to promote Softbank, he might not be the best CEO to research any signals from.\n",
    "\n",
    "## 10. Hiroshi Mikitani of Rakuten / 三木谷浩史 (楽天株式会社)\n",
    "\n",
    "During the same period, **Hiroshi Mikitani**, the founder and chairman of **Rakuten** shared around 230 tweets to his 1.1M followers. The Japanese entrepreneur is carefully sharing about his ventures (**Lyft, Viber, Pinterest**), Rakuten’s financial results, his baseball team **Tohoku Rakuten Golden Eagles**, partnership in sports (**FC Barcelona, Golden State Warriors**) and meditation. As a digital-savvy, Hiroshi Mikitani secured an investment deal worth `150` billion yen (`$1.4` billion) from the Japan Post and Tencent Investment for an `8%` stake in Rakuten. More on the social media usage, the Japanese CEO displays a clear pattern in his digital communication."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24477a68",
   "metadata": {},
   "source": [
    "## 11. Governor of the Bank of Japan, Haruhiko Kuroda / 第31代日本銀行総裁、黒田 東彦\n",
    "After **Jerome Powell** of the US Federal Reserve, **Haruhiko Kuroda** of Bank of Japan is probably one of the most carefully listened bank governors. The 31st Japanese governor is not on Twitter, but the Bank of Japan is constantly sharing great insights both in Japanese and in English.\n",
    "\n",
    "Yet, not fully... let's look at the RSS feed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "4924001a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import feedparser\n",
    "jarawrss = ['https://www.boj.or.jp/rss/whatsnew.xml']\n",
    "enrawrss = ['https://www.boj.or.jp/en/rss/whatsnew.xml',]\n",
    "\n",
    "feedsja = []\n",
    "for url in jarawrss:\n",
    "    feedsja.append(feedparser.parse(url))\n",
    "\n",
    "postsja = [] \n",
    "for feed in feedsja:\n",
    "    for post in feed.entries:\n",
    "        postsja.append((post.published, post.title, post.link))\n",
    "\n",
    "JA = pd.DataFrame(postsja, columns=['JA_Date','JA_Title','JA_Link']) \n",
    "\n",
    "feeds = []\n",
    "for url in enrawrss:\n",
    "    feeds.append(feedparser.parse(url))\n",
    "\n",
    "posts = []\n",
    "for feed in feeds:\n",
    "    for post in feed.entries:\n",
    "        posts.append((post.published,post.title, post.link ))\n",
    "\n",
    "EN = pd.DataFrame(posts, columns=['EN_Date','EN_Title', 'EN_Link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "e7619ea1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>JA_Date</th>\n",
       "      <th>JA_Title</th>\n",
       "      <th>JA_Link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mon, 14 Jun 2021 17:00:00 +0900</td>\n",
       "      <td>日本銀行による国庫短期証券の銘柄別買入額</td>\n",
       "      <td>http://www.boj.or.jp/statistics/boj/other/tmei...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mon, 14 Jun 2021 17:00:00 +0900</td>\n",
       "      <td>日本銀行が保有する国債の銘柄別残高</td>\n",
       "      <td>http://www.boj.or.jp/statistics/boj/other/mei/...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mon, 14 Jun 2021 11:00:00 +0900</td>\n",
       "      <td>（金研ニュースレター）2021年国際コンファランス</td>\n",
       "      <td>http://www.boj.or.jp/announcements/release_202...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mon, 14 Jun 2021 10:00:00 +0900</td>\n",
       "      <td>営業毎旬報告（6月10日現在）</td>\n",
       "      <td>http://www.boj.or.jp/statistics/boj/other/acma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Mon, 14 Jun 2021 10:00:00 +0900</td>\n",
       "      <td>金融広報関連事務担当者の募集について</td>\n",
       "      <td>http://www.boj.or.jp/announcements/release_202...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           JA_Date                   JA_Title  \\\n",
       "0  Mon, 14 Jun 2021 17:00:00 +0900       日本銀行による国庫短期証券の銘柄別買入額   \n",
       "1  Mon, 14 Jun 2021 17:00:00 +0900          日本銀行が保有する国債の銘柄別残高   \n",
       "2  Mon, 14 Jun 2021 11:00:00 +0900  （金研ニュースレター）2021年国際コンファランス   \n",
       "3  Mon, 14 Jun 2021 10:00:00 +0900            営業毎旬報告（6月10日現在）   \n",
       "4  Mon, 14 Jun 2021 10:00:00 +0900         金融広報関連事務担当者の募集について   \n",
       "\n",
       "                                             JA_Link  \n",
       "0  http://www.boj.or.jp/statistics/boj/other/tmei...  \n",
       "1  http://www.boj.or.jp/statistics/boj/other/mei/...  \n",
       "2  http://www.boj.or.jp/announcements/release_202...  \n",
       "3  http://www.boj.or.jp/statistics/boj/other/acma...  \n",
       "4  http://www.boj.or.jp/announcements/release_202...  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JA.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "fbd9a142",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EN_Date</th>\n",
       "      <th>EN_Title</th>\n",
       "      <th>EN_Link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mon, 14 Jun 2021 17:00:00 +0900</td>\n",
       "      <td>T-Bills Purchased by the Bank of Japan</td>\n",
       "      <td>http://www.boj.or.jp/en/statistics/boj/other/t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mon, 14 Jun 2021 17:00:00 +0900</td>\n",
       "      <td>Japanese Government Bonds Held by the Bank of ...</td>\n",
       "      <td>http://www.boj.or.jp/en/statistics/boj/other/m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mon, 14 Jun 2021 11:00:00 +0900</td>\n",
       "      <td>(IMES Newsletter) 2021 BOJ-IMES Conference</td>\n",
       "      <td>http://www.boj.or.jp/en/announcements/release_...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mon, 14 Jun 2021 10:00:00 +0900</td>\n",
       "      <td>Bank of Japan Accounts (June 10)</td>\n",
       "      <td>http://www.boj.or.jp/en/statistics/boj/other/a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thu, 10 Jun 2021 16:30:00 +0900</td>\n",
       "      <td>Basic Figures on Fails (May)</td>\n",
       "      <td>http://www.boj.or.jp/en/statistics/set/bffail/...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           EN_Date  \\\n",
       "0  Mon, 14 Jun 2021 17:00:00 +0900   \n",
       "1  Mon, 14 Jun 2021 17:00:00 +0900   \n",
       "2  Mon, 14 Jun 2021 11:00:00 +0900   \n",
       "3  Mon, 14 Jun 2021 10:00:00 +0900   \n",
       "4  Thu, 10 Jun 2021 16:30:00 +0900   \n",
       "\n",
       "                                            EN_Title  \\\n",
       "0             T-Bills Purchased by the Bank of Japan   \n",
       "1  Japanese Government Bonds Held by the Bank of ...   \n",
       "2         (IMES Newsletter) 2021 BOJ-IMES Conference   \n",
       "3                   Bank of Japan Accounts (June 10)   \n",
       "4                       Basic Figures on Fails (May)   \n",
       "\n",
       "                                             EN_Link  \n",
       "0  http://www.boj.or.jp/en/statistics/boj/other/t...  \n",
       "1  http://www.boj.or.jp/en/statistics/boj/other/m...  \n",
       "2  http://www.boj.or.jp/en/announcements/release_...  \n",
       "3  http://www.boj.or.jp/en/statistics/boj/other/a...  \n",
       "4  http://www.boj.or.jp/en/statistics/set/bffail/...  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EN.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "e1ca51f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bank of Japan - RSS feed - Articles in Japanese (89, 3)\n",
      "Bank of Japan - RSS feed - Articles in English (64, 3)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Bank of Japan - RSS feed - Articles in Japanese\",JA.shape)\n",
    "print(f\"Bank of Japan - RSS feed - Articles in English\",EN.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6ba8c9a",
   "metadata": {},
   "source": [
    "The difference in the RSS feed of BOJ is the perfect illustration for the asymmetric of information due to language. The extremely interesting research paper about the **\"Digital Transformation of Japanese Banks (わが国の銀行におけるデジタル・トランスフォーメーション)\"** was published in March 29th in Japanese and May 31st 2021 in English. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "200ff778",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #ffeb80; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    新型コロナウイルス\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">EVENT</span>\n",
       "</mark>\n",
       "感染症対応金融支援特別オペにおける貸付対象先公募の結果について</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "research = JA['JA_Title'][85]\n",
    "doc8 = nlp_bespoke(research)\n",
    "displacy.render(doc8,style=\"ent\",jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c13f0da8",
   "metadata": {},
   "source": [
    "And yes, my custom NER model is still working on the RSS feed of the BOJ. For legal reason, I will not go deeper on this notebook. Overall, the **Bank of Japan** is sharing 30% more reports, statitics, research/studies, and announcements in Japanese, than in English."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e9b53ed",
   "metadata": {},
   "source": [
    "## 13. Final thoughts on NER in Japanese\n",
    "\n",
    "Fifteen years ago, French was difficult to process using C++, Pascal, and Object Pascal. Eight years ago, Japanese was still a nightmare. In recent years, it became really easy to do NLP using a framework in multiple languages.\n",
    "\n",
    "SpaCy is fast, powerful, stable, and perfect for production. Ginza could be leveraged for Japan in a less domain-specific task. I achieved pretty amazing results with my custom model based on a small training datasets.\n",
    "\n",
    "In the financial sector, a bespoke NER model needs to include corporate entities, subsidiaries, titles (CEO, CFO), boards of directors (for financial reports), tickers, Bloomberg shortcuts (TPX/JSDA), macro phenomenon, currencies, products, business partnerships, and financial keywords such as “赤字” for deficit or “倒産” for bankruptcy.\n",
    "\n",
    "In Japanese language, the tokenization can play a huge role in the final results. Wrong tokenization often leads to loss in both syntax and semantic.\n",
    "\n",
    "Coming from a linguistic background, upcoming data scientists are focusing too much on the latest neural model or scores, when it’s important to have good labeling in the context and a stable model.\n",
    "\n",
    "The optimal results in NLP are often relying on linguistic basics, proficiency in the language, and the knowledge of one particular sector. I will also argue that Japanese is mandatory but incorporating correct English can create better NER models in Japanese.\n",
    "\n",
    "Thank you for reading!\n",
    "\n",
    "Please feel free to contact me if you have any questions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22de8a50",
   "metadata": {},
   "source": [
    "**Akim Mousterou**\n",
    "\n",
    "***\n",
    "**Disclaimer**: *None of the content published on this notebook constitutes a recommendation that any particular security, portfolio of securities, transaction, or investment strategy is suitable for any specific person. None of the information providers or their affiliates will advise you personally concerning the nature, potential, value, or suitability of any particular security, portfolio of securities, transaction, investment strategy, or other matter.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}