{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# ตัวอย่างการใช้ `pandas`\n", "\n", "Example : Blog gender dataset\n", "\n", "เราต้องการเตรียมข้อมูลสำหรับการสร้างเครื่องแยกแยะว่า Blog post นั้นเขียนโดยผู้เขียนที่เป็นผู้ชายหรือผู้หญิง" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "import pandas as pd\n", "import nltk\n", "from sklearn.model_selection import train_test_split\n", "\n", "blog_data = pd.read_csv('https://attapol.github.io/programming/data/blog-gender-dataset.csv', \n", " names=['blog_post', 'gender'])\n", "blog_data['length'] = blog_data['blog_post'].apply(str).apply(len)\n", "blog_data = blog_data[blog_data['length'] > 80]\n", "blog_data['tokenized_text'] = blog_data['blog_post'].apply(lambda x: '|'.join(nltk.word_tokenize(str(x))))\n", "blog_data['label'] = blog_data['gender'].apply(lambda x: x.upper().strip())\n", "\n", "blog_data = blog_data[['label', 'tokenized_text']]\n", "train, the_rest = train_test_split(blog_data, test_size=0.4)\n", "dev, test = train_test_split(the_rest, test_size=0.5)\n", "\n", "train.to_csv('train-blog-gender-dataset.csv', index=False)\n", "dev.to_csv('dev-blog-gender-dataset.csv', index=False)\n", "test.to_csv('test-blog-gender-dataset.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 1. Load and clean raw data" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "scrolled": true, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textgender
0Long time no see. Like always I was rewriting...M
1Guest Demo: Eric Iverson’s Itty Bitty Search\\...M
2Who moved my Cheese??? The world has been de...M
3Yesterday I attended a biweekly meeting of an...M
4Liam is nothing like Natalie. Natalie never w...F
5In the EU we have browser choice, but few know...M
6Hmmm.. I really didn't wanna update my blog ti...F
7happy teachers day..!! who is celebrating..??\\...M
8We watch movies. And we see what the camera in...F
9Cooking! May be the title of the blog is a gi...F
\n", "
" ], "text/plain": [ " text gender\n", "0 Long time no see. Like always I was rewriting... M\n", "1 Guest Demo: Eric Iverson’s Itty Bitty Search\\... M\n", "2 Who moved my Cheese??? The world has been de... M\n", "3 Yesterday I attended a biweekly meeting of an... M\n", "4 Liam is nothing like Natalie. Natalie never w... F\n", "5 In the EU we have browser choice, but few know... M\n", "6 Hmmm.. I really didn't wanna update my blog ti... F\n", "7 happy teachers day..!! who is celebrating..??\\... M\n", "8 We watch movies. And we see what the camera in... F\n", "9 Cooking! May be the title of the blog is a gi... F" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "data = pd.read_csv('https://attapol.github.io/programming/data/blog-gender-dataset.csv', \n", " names=['text', 'gender'])\n", "data.head(10)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### ตรวจสอบความถูกต้องของข้อมูล\n", "\n", "ลองสุ่มขึ้นมาดูเรื่อย ๆ ว่ามีอะไรผิดปกติมั้ย" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[\"Ahem!This hereby serves as a final reminder that I've moved my blog to my new website, johnsellers.com. Please update your links -- if you even bothered to bookmark this nonsense in the first place, that is.Also, this hereby serves as yet another excuse for me to publish a video of a monkey riding a bicycle. After five years and 454 posts, I am hereby and forevermore moving this blog over to the much-simpler-for-me johnsellers.com.Please update your links.This is your new RSS feed.Enjoy the silence.WHY I'M ANGRY TODAYHoagie crumbs stuck in my sweater! Here's the interview I did with John Cleese.WHY I'M ANGRY TODAYWhy is Chase Bank messing with me? They be frontin'! I am interviewing Lost's Jorge Garcia today, who it turns out has a pretty sweet blog called Dispatches from the Island. Here's my favorite post.In the next few days, I'm going to post a list of the top ten little things that annoyed me this year. And in the next few weeks, I'm going to be moving this blog over to johnsellers.com -- but only after I figure out how to do that. But this is what it's going to look like.Finally, I'm going to stop titling each of my posts after random favorite songs. Instead, I shall name each post after my favorite word in said post.WHY I'M ANGRY TODAYWhy hasn't teleportation been invented yet? That way I could get to the post office without a hassle. And here is my interview with Benicio Del Toro.WHY I'M ANGRY TODAYGoddamn cold. I have three Q+As out right now. In order of how well I \"]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.sample(n=1)['text'].to_list()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "ดูความยาวเฉลี่ยของข้อความดูว่ามันอยู่ช่วงที่โอเคมั้ย\n", "\n", "(ให้สังเกตวิธีการเพิ่ม คอลัมน์ใหม่ให้กับ `DataFrame`)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(3232, 2)" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.shape" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "data['text length'] = data['text'].apply(lambda x: len(str(x)))" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 3232.000000\n", "mean 2343.352104\n", "std 4579.572823\n", "min 3.000000\n", "25% 570.000000\n", "50% 1049.000000\n", "75% 1879.250000\n", "max 32714.000000\n", "Name: text length, dtype: float64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['text length'].describe()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textgendertext length
0Long time no see. Like always I was rewriting...M954
1Guest Demo: Eric Iverson’s Itty Bitty Search\\...M1877
2Who moved my Cheese??? The world has been de...M5983
3Yesterday I attended a biweekly meeting of an...M1132
4Liam is nothing like Natalie. Natalie never w...F1139
\n", "
" ], "text/plain": [ " text gender text length\n", "0 Long time no see. Like always I was rewriting... M 954\n", "1 Guest Demo: Eric Iverson’s Itty Bitty Search\\... M 1877\n", "2 Who moved my Cheese??? The world has been de... M 5983\n", "3 Yesterday I attended a biweekly meeting of an... M 1132\n", "4 Liam is nothing like Natalie. Natalie never w... F 1139" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "7573714" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['text length'].sum()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "เอ ทำไมบางข้อความมันสั้นจัง ลองลงไปตรวจหน่อย\n", "\n", "(สังเกตวิธีการเลือกแค่บางแถว)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textgendertext length
142NaNNaN3
999NaNNaN3
1010NaNNaN3
1024NaNNaN3
1471NaNNaN3
1521NaNM3
1728i love these vitamins when they were from AARP...M85
\n", "
" ], "text/plain": [ " text gender text length\n", "142 NaN NaN 3\n", "999 NaN NaN 3\n", "1010 NaN NaN 3\n", "1024 NaN NaN 3\n", "1471 NaN NaN 3\n", "1521 NaN M 3\n", "1728 i love these vitamins when they were from AARP... M 85" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "is_too_short = data['text length'] < 100\n", "data[is_too_short]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "สร้าง `DataFrame` ขึ้นมาใหม่ที่สะอาดกว่าเดิม" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(3226, 3)\n", "(3232, 3)\n" ] } ], "source": [ "new_data = data[data['text length'] > 80]\n", "print(new_data.shape)\n", "print(data.shape)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### เช็คการกระจายตัวของ Label\n", "แปลว่า เช็คว่า label แต่ละขนิดเกิดขึ้นมากน้อยกี่ครั้ง ดูด้วยว่ามี label แปลกปลอมมามั้ย" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "data": { "text/plain": [ "M 1546\n", "F 1390\n", " F 153\n", " M 126\n", "m 5\n", "f 4\n", "F 1\n", " M 1\n", "Name: gender, dtype: int64" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_data['gender'].value_counts()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "เปลี่ยนตัวเล็กให้เป็นตัวใหญ่ และเอา space หน้าหลังออกให้หมด\n", "\n", "เสร็จแล้วสร้างเป็น column ใหม่แยกกันออกมา\n" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/plain": [ "M 1678\n", "F 1548\n", "Name: gender, dtype: int64" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_data['gender'].apply(lambda x: x.upper().strip()).value_counts()" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/te/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n", "A value is trying to be set on a copy of a slice from a DataFrame.\n", "Try using .loc[row_indexer,col_indexer] = value instead\n", "\n", "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", " \"\"\"Entry point for launching an IPython kernel.\n" ] } ], "source": [ "new_data['label'] = new_data['gender'].apply(lambda x: x.upper().strip())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 2. Tokenize, tag, and parse data\n" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: nltk in /Users/te/opt/anaconda3/lib/python3.7/site-packages (3.6.7)\n", "Requirement already satisfied: click in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from nltk) (8.0.3)\n", "Requirement already satisfied: regex>=2021.8.3 in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from nltk) (2021.11.10)\n", "Requirement already satisfied: joblib in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from nltk) (1.1.0)\n", "Requirement already satisfied: tqdm in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from nltk) (4.62.3)\n", "Requirement already satisfied: importlib-metadata in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from click->nltk) (4.8.2)\n", "Requirement already satisfied: typing-extensions>=3.6.4 in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from importlib-metadata->click->nltk) (3.10.0.2)\n", "Requirement already satisfied: zipp>=0.5 in /Users/te/opt/anaconda3/lib/python3.7/site-packages (from importlib-metadata->click->nltk) (3.6.0)\n" ] } ], "source": [ "!pip install nltk" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/te/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n", "A value is trying to be set on a copy of a slice from a DataFrame.\n", "Try using .loc[row_indexer,col_indexer] = value instead\n", "\n", "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", " \n" ] } ], "source": [ "import nltk\n", "new_data['tokenized_text'] = new_data['text'].apply(lambda x: '|'.join(nltk.word_tokenize(x)))" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textgendertext lengthlabeltokenized_text
0Long time no see. Like always I was rewriting...M954MLong|time|no|see|.|Like|always|I|was|rewriting...
1Guest Demo: Eric Iverson’s Itty Bitty Search\\...M1877MGuest|Demo|:|Eric|Iverson|’|s|Itty|Bitty|Searc...
2Who moved my Cheese??? The world has been de...M5983MWho|moved|my|Cheese|?|?|?|The|world|has|been|d...
3Yesterday I attended a biweekly meeting of an...M1132MYesterday|I|attended|a|biweekly|meeting|of|an|...
4Liam is nothing like Natalie. Natalie never w...F1139FLiam|is|nothing|like|Natalie|.|Natalie|never|w...
\n", "
" ], "text/plain": [ " text gender text length \\\n", "0 Long time no see. Like always I was rewriting... M 954 \n", "1 Guest Demo: Eric Iverson’s Itty Bitty Search\\... M 1877 \n", "2 Who moved my Cheese??? The world has been de... M 5983 \n", "3 Yesterday I attended a biweekly meeting of an... M 1132 \n", "4 Liam is nothing like Natalie. Natalie never w... F 1139 \n", "\n", " label tokenized_text \n", "0 M Long|time|no|see|.|Like|always|I|was|rewriting... \n", "1 M Guest|Demo|:|Eric|Iverson|’|s|Itty|Bitty|Searc... \n", "2 M Who|moved|my|Cheese|?|?|?|The|world|has|been|d... \n", "3 M Yesterday|I|attended|a|biweekly|meeting|of|an|... \n", "4 F Liam|is|nothing|like|Natalie|.|Natalie|never|w... " ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_data.head()" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "new_data = new_data[['tokenized_text', 'gender']]" ] }, { "attachments": { "image.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABLgAAAA8CAYAAABlywOLAAAaz0lEQVR4Ae2dCewdUxvGW2srKSEkCIktraD9bLHGWkEbYgliTUuEENSSUFskliZKQ8VSgoRYSxW1xBJLbVW6N5Taqq3aWlSptjhfnvPlTGY5939neefcO+/3TPLvzJyZzjn3+b3v3HOeO0svE3CaO3duwNpYVSgFyDWU0mHrIdeweoeqjVxDKR22HnINq3eo2sg1lNJh6yHXsHqHqo1cQykdth5yDat3qNrINZTSYesB114hq2QghVQ7XF3kGk7rkDWRa0i1w9VFruG0DlkTuYZUO1xd5BpO65A1kWtItcPVRa7htA5ZE7mGVDtcXeQaTuuQNYErDa6QiiutiycInWDJlVx1KqDzUzFfyVWnAjo/FfOVXHUqoPNTMV/JVacCOj8V8pUGl062QT8VT/xB5Q5WGbkGkzpoReQaVO5glZFrMKmDVkSuQeUOVhm5BpM6aEXkGlTuYJWRazCpg1ZErkHlDlYZuNLgCia33op4gtDJllzJVacCOj8V85VcdSqg81MxX8lVpwI6PxXzlVx1KqDzUyFfaXDpZBv0U/HEH1TuYJWRazCpg1ZErkHlDlYZuQaTOmhF5BpU7mCVkWswqYNWRK5B5Q5WGbkGkzpoReQaVO5glYErDa5gcuutiCcInWzJlVx1KqDzUzFfyVWnAjo/FfOVXHUqoPNTMV/JVacCOj8V8lXE4Pr222/NM888Yz744AOzcuXKlmrxBNFSGvENeZm0q3j27Nlm8uTJZvXq1S13JdeW0ohvqMp18eLF5sUXXzRPPfWUmTVrFrmKEyp3wKpcFyxYYF544QUzfvx48/HHH5tVq1a1bAjztaU04huqcnUNWrNmjZk+fbqZOnWqWbZsmStOzMk1IUetK1W4fvHFF2bevHmZP5T7JnL1qVJPWRWurkV///23+fzzz82kSZPMhAkT7Pfsv//+6zZHc3KNpKh9oSzXr776KpOn8dz97LPPTJotudaOM6qgLFd3gBUrVtjv1CeffNK89NJLZuHChW5TZk6uGUlqK6jKFT7EtGnTDLhi/Lp8+fKWbSXXltJ07Ya8vkQlg2vOnDlmu+22M7169Yr+1l13XTNq1CivMAwkryyihUWZtKr8p59+MqeffnrEdebMma12NeTaUhqxDVW5osN92GGHmd69e0dMkbcDBgwwH374obed5OqVRbSwKld0tg866KAEU3Ddfvvtzfvvv+9tK7l6ZREtrMo13ZgRI0ZEjE866aT0ZrtOrl5ZRAurcsUgKt5fSi/jB4j0RK5pReTXq3J1LZo4caLZdtttM4yHDx/udonm5BpJUdtCFa5//PFHpr+UzlesT5kyJdF+ck3IUctKFa6uQXfccYfZaKONErmK/jHGPX/99ZfbLZqTayRFbQtVucJsvvPOO03fvn0TXDfddFNrdvkaTq4+VbqzrKgvUdrgwsAKQYMT/F577WWuvfZac8YZZ5i11lrLlo0dOzajEAMpI4loQRkmvgY88sgjEVv3hU6Dy6dUmDIJrsOGDbN5ic73mWeeaU444QSz9tpr27KtttrK/P7775kPw3zNSCJaIMEVLN05+JprrjGXXXZZ1GkDV98VteQqijFzMAmu8YPi1yqXq2CN3PVN5OpTRa5MgiuusATDbbbZxgwdOjTxd9xxxzFf5XDlPpIEV1SGqwVcf2nw4MHmhhtuMLfeeqs5/vjjzXnnnZdpD/M1I4loQVWuGCyD3RFHHJH522effSLWM2bMSLSbXBNyiK9U5YoG4Wp35Cq+V4899lgDs+vyyy83G2ywgS0fOXJkpt3kmpFEtECC69133235wYc4++yzzS233GIOOeQQW4aLb9K5ig9ArqIYaztYGV+itMF1wQUX2KA59NBDzT///BN9qBtvvNGW77DDDlGZW2AgOSXqmZdhkm4JbjPFib9fv35mzJgx0WCZBldaqXDrElzvu+8+88ADDyRyNd4hx5UF6Yn5mlZEdl2C62OPPWZvOY237L333rM5jDz+6KOP4pvsMrlmJBEtkOAab5C7Qg9XYILpiSeeGN8cLZNrJEUtCxJcncF1xRVX5G4jueaWqtSOElx//fVX22dCfo4ePTpXO8g1l0yld5Lg2qpy9I3BeqeddsrsQq4ZSUQLJLi6u1PSV0Pfddddluuuu+6aaTO5ZiQRLajKFT6Eu+gGhqWbcMs4fArkKy7CSU/kmlak+9bL+hKlDC48j8kF0jvvvJNQ45dffjHrrbeeDSbcAxufGEhxNWSXyzJJt2L+/PkGJ5olS5bYTZtttpllSYMrrVSYdSmuvtbiF8o+ffpYvuPGjcvswnzNSCJWUCdXfKG7K2lfe+21TJvJNSOJWIE018cff9zm54EHHmgNahpcYqgKHUiKKw2uQrLXvrMUVwyokJu4sifvxPNwXqWK7yfF1VcznoeIKzDB+5577snsQq4ZScQKpLi6H4tuu+22RNvuvfdey3XQoEGJcqyQa0YSsQIJrnheHnJynXXWyTyD9umnn7bbttxyy0ybyTUjSdcVlPUlShlceLAiAgn3uSIw09PAgQPtdnTO4xMDKa6G7HJZJu1aQYOrnUL1bq+LK1qNZ0w4IwRfAOmJ+ZpWRG69Tq6vvPKKPf+C7aJFizKNJteMJGIFklxx2zBuM8VtFHghxIMPPmi58gouMVy5DyTFlQZXbsmD7CjFFbc5oU+MK6XzTjwP51Wq+H5SXH01uyvfN954Y9uHSu9DrmlF5NaluF555ZU2X2FU4oHmmHAV5m677WbLsT09kWtaEbl1Ca7uzgXkZXoCY5yf8QfO8Ylc42o0YzmvL1HK4MJVWwgUPM/HNx111FF2Ox72Fp8YSHE1ZJfLMmnXiryB1O443F5Ogbq4ojW4vQ15jF88fvzxx0wDma8ZScQKJLn++eef1sjCAzpxib27uhZXYvomcvWpIlMmyRW3sSE/L7zwQts4GlwyjMocRYqrM7gwsBoyZIh9/guem/fcc8+1bBbztaU0lTdIcXUvW8KVXOCJK0T2228/g4fLt2JLrpXxtTyAFFdfBXjmMM7LeGaTbyJXnyoyZVJcly5davr37285or90/fXXRy+HQN7ymbQyvPIeRYIr3iSOvMRf+o3E6CO7bd98802iWczXhByNWMnrS5QyuNzlfr7LOKEO7mtGMN10000JsRhICTlEV8oyadeIvIHU7jjcXk6BurjiLTF4Th7yFM8j8E3MV58qMmWSXM8///zoy9t9iR955JHeNwGh9eQqw9B3FCmu+EUTt/rj/Ivb/jHR4PIpHqZMiuukSZMyuepy9oADDjDoiKcn5mtaEbl1Ka7pt7HhgcaOK+a8IkSOWZ4jSXFN1/Xuu+9arriqFgNq38R89akiUybJFT/q4pwbz9P/DNrVew5G68lVhqHvKBJc8QwuPBMPPPfcc0/z9ttvG7w9Hi8UOProoyPOX375ZaIJ5JqQoxEreX2JUgbXhAkTbLDsuOOOXjGcwTVq1KjEdgZSQg7RlbJM2jUibyC1Ow63l1OgLq54qxO+CPDrFV696puYrz5VZMokuT777LP2jTEnn3yy2X333aMvcjwo9Ycffsg0mFwzkogVSHGFQYn8vP/++6O20eCKpAi+IMUVnXD8ggzTcsWKFbYDjrft4Spa8L700kszn435mpFErECK6/rrr2/54fZhvNgDnJcvX27cVZjgmzZEyFUMY+ZAUlzTB8ZbFZGnrd5ki/3JNa2a3LokV7wQzeWtM0bAFqYX+01yzPIcSYrrm2++aTbZZBObo2Dp/nr37h0tL168ONEk5mtCjkas5PUlShlcb731lg2WLbbYwiuGu0Ux3jnHjgwkr1wihWWZtKs8byC1Ow63l1OgDq5jx461+YsrRN54442WDWO+tpSm8oY6uLpGTZ48OXqj17Bhw1xxNCfXSArxBQmu7iof3AqDF0G4iQaXUyL8XIJrT62++uqr7Tl5wIABmd2YrxlJxAqkuLoruKZMmZJoG4wuPEcPA62JEycmtpFrQg7RFSmu8Ubhyg/3zFLcUtVqItdWylQvl+J61VVXRWNYx/LVV1+NcnWPPfYweFlPfCLXuBqyy1Jc0arvvvvOvvzhoosusi9Mu/322w3OyzgH44cG3L0Sn8g1rkYzlvP6EqUMLgQEggWXYa9cuTKjiHPDn3/++cQ2BlJCDtGVskzaNSJvILU7DreXU0Ca6xNPPGE7afhF49FHH+2xUczXHuWptFGaa7oxI0eOtOdoPOsnPZFrWhG5dQmu5557rmWH71j8wuz+3ODKlYNxfCLXuBqyyxJce2oRbqMA1w033DCzG7lmJBErkOLq3qoHjulp7733tmwfeuihxCZyTcghuiLFNd4oDJiRo7hKuqeJXHtSp9o2Ca7Lli2LrpidOnVqokF4NIC7qgumS3wi17gasssSXHtqEd4mjtzFLajpiVzTinT/el5fopTBhedE9OvXzwbMyy+/nFBj4cKFBoNnmF/pW58YSAmpRFfKMmnXiLyB1O443F5OAUmuuDLEPRsEv2q0m5iv7RQqv12Sq68VeP4hvtDxnLX0RK5pReTWJbgiN3F7afrPXQmC715sS7/inFzlOKaPJME1fcz4Oh5OjnzFW7zSE7mmFZFbl+I6dOhQyy/9rC1cBYJX04MtB8xy3NodSYqrqwe3FLsxT9qodPu4OfPVKSE/l+DqrubBHQy+CzTc9yyeCxWfyDWuhuyyBNeeWuSewYUrpdMTuaYV6f71vL5EKYMLH//UU0+1X9p444Q7SeDL/JRTTrHleG1yemIgpRWRXS/CBJdbn3POOQbP/+hpyhtIPR2D26opIMEV96b36dPH5mb65Q+tWsd8baWMTLkEV7wpcdq0aYkG4RkDW2+9tWWNOtITuaYVkV2X4OprEW9R9KkSrkyCKwbHv/32W6LReOPT5ptvbvP1kksuSWzDCvM1I4logQRXmM0wsTA4xi0ybhozZowtx3Nh8Eyu+ESucTXklyW4ulbdfPPNliPydNWqVa7YOydXryxihVW5fv/995Yl8nXcuHGJdr3++uvRtk8++SSxjVwTcoivVOWKBqVvP0TZddddZ5n27dvXpJ+/he3kKo6y9gPm9SVKG1x4kKb7RQOvSIaxtcsuu9hAwoOrZ86cmfmQDKSMJKIFRZiMHj3assI9yatXr47asWTJEvuK68GDBxv84W0x+CLAWymw7ns7G7lG8tWyIMEVL4QAR/wNHDgw84fn/axZsybRfnJNyCG+UpUrflBwt62BKToI+GHBnZfxXJivv/46025yzUgiWlCVa6vG0OBqpUyY8qpc8Ss1zr/oaB9++OFm+PDhZsiQIdFVtbjakq+nD8MyXktVrjgWbntyV37gii28mfjggw+OvnPTA2n8H56H4xTklyW4olXoHzu2GCy3m8i1nULVtktwPeaYY2xu4m6jQYMGmbPOOsvsu+++0XgH/aj48y/RYnKtxq3d/5bguvPOO9s3KeK79bTTTjPwJvCdC87jx4/3NoFcvbJ0VWFZX6K0wYVPj0s9cXJwJgjuXd5///3N7NmzveIwkLyyiBbmZeJey9q/f/9E/fPmzbMnA2eGpOe4xQ0d9fhErnE16lmuyhUGSJplfB2DLhpc9bDr6ahVueJFHvhSj7OEaY1LstO/QLp2MF+dEvXNq3L1tezhhx+2nPFjkm8iV58qsmVVuY4YMcK+uTaerzCk8Xyfn3/+2dtYcvXKIlpYlSsaM2fOHDtIdmwxqIJpmX64vGs4uTol6ptLcHVX9SBPMdBqN5FrO4Wqb6/KFVdTXnzxxdGPgS5n8QxElKevtkSLybU6t3ZHqMoVzy91P/qCKbwJ3GXmXiTgq59cfap0V1lZX6KSweUkwOuu8eXOS3edIp2f52GydOlS+zrrqq3lCaKqgvn/P7nm16pJe1blilzGDwvz58/PGJVpHZivaUXqW6/KNd0y3N6WfruT24dcnRL1z6twxZUBCxYsMDNmzEjcztaq1eTaShn58ipcXWtwLp41a1bmVlS33c3J1SlR/7wqV5x3241v3KcgV6dE/fOqXPGW00WLFtk7jnDrYvqqrfgnINe4GvUuV+GKq6DhR3z66ae5cpZc62XZqaODq4jBlfcDMJDyKtWs/ci1WbzytpZc8yrVrP3ItVm88raWXPMq1az9yLVZvPK2llzzKtWs/ci1WbzytpZc8yrVrP3ItVm88rYWXGlw5VWL+7VUgCeIltI0egO5Nhpfy8aTa0tpGr2BXBuNr2XjybWlNI3eQK6Nxtey8eTaUppGbyDXRuNr2XhybSlNozeAKw2uRiPsjsbzBNEdHKRbQa7SinbH8ci1OzhIt4JcpRXtjuORa3dwkG4FuUor2h3HI9fu4CDdCnKVVrQ7jkeu3cFBuhXgSoNLWtX/w+PxBKETOrmSq04FdH4q5iu56lRA56divpKrTgV0firmK7nqVEDnp0K+0uDSyTbop+KJP6jcwSoj12BSB62IXIPKHawycg0mddCKyDWo3MEqI9dgUgetiFyDyh2sMnINJnXQisg1qNzBKgPXXvgn1J97FSvnvezr3qkDdWAMMAYYA4wBxgBjgDHAGGAMMAYYA4wBxgBjgDEgEAPB7DRjrKkzc/p0wz9dGiARyVQXU/AkV31MyVUnU3IlV34HNysG8P3628fP8U+ZBuSqM6bB9evrD+OfMg3IVWdMg2vQWxRRITthzeqE5eFFrvqYcsCskym5kmueczr36Z444fdr97CQzAtwpcGlzwwhV31MkafgSoNLnxlCrvqYIk/BlQYXryirbDoikCQ7fjxWd3ToybU7OEjnA7mSq3RM8Xj1xRTztT5tOxm34EqDS58ZQq76mNLg0mmCOCOExqU+vjgP0+CiwVXZnGIHXG8HvJMDANZdT1wxX+vRtdPxSq7k2ukYZP35YxD5SoNLnxlCrvqY0uDSZ4A4Uwv56pY518MZXGlw0eCiwcUY8MYAThAcsOQfsDRFK3LVxxSxR67k2pRzENv5v3ylwaXPDMF5mFx1cqUBoscAcSyRr26Zcz18wZUGF82NyiYGB1YcWHHA0pwYYL42h1WRvCJXci0SL9y3s/GCfKURotMIIVedXGmA6DFAHEuch90y53r4gisNLhpcNLgYA94Y4IC5swOgugag5EqudcUWjysfW8xXeU27IU7BlUaITiOEXHVypQGixwBxLHEedsuc6+ELrjS4aG54zY0iHUB2wPV2wIvEAfdtRhwwX5vBqWg+kSu5Fo0Z7t+5mEG+0gjRaYSQq06uNED0GCCOJc7DbplzPXzBlQYXDS4aXIwBbwxwwNy5wU+dA09yJdc644vHlo0v5qusnt0Sn+BKI0SnEUKuOrnSANFjgDiWOA+7Zc718AVXGlw0N7zmRpFOIDvgejvgReKA+zYjDpivzeBUNJ/IlVyLxgz371zMIF9phOg0QshVJ1caIHoMEMcS52G3zLkevuBKg4sGFw0uxoA3Bjhg7tzgp86BJ7mSa53xxWPLxhfzVVbPbolPcKURotMIIVedXGmA6DFAHEuch90y53r4gisNLpobXnOjSCeQHXC9HfAiccB9mxEHzNdmcCqaT+RKrkVjhvt3LmaQrzRCdBoh5KqTKw0QPQaIY4nzsFvmXA9fcKXBRYOLBhdjwBsDHDB3bvBT58CTXMm1zvjisWXji/kqq2e3xCe40gjRaYSQq06uNED0GCCOJc7DbplzPXzBlQYXzQ2vuVGkE8gOuN4OeJE44L7NiAPmazM4Fc0nciXXojHD/TsXM8hXGiE6jRBy1cmVBogeA8SxxHnYLXOuhy+40uCiwUWDizHgjQEOmDs3+Klz4Emu5FpnfPHYsvHFfJXVs1viE1xphOg0QshVJ1caIHoMEMcS52G3zLkevuBKg4vmhtfcKNIJZAdcbwe8SBxw32bEAfO1GZyK5hO5kmvRmOH+nYsZ5CuNEJ1GCLnq5EoDRI8B4ljiPOyWOdfDF1xpcNHgosHFGPDGAAfMnRv81DnwJFdyrTO+eGzZ+GK+yurZLfEJrjRCdBoh5KqTKw0QPQaIY4nzsFvmXA9fcKXBRXPDa24U6QSyA663A14kDrhvM+KA+doMTkXziVzJtWjMcP/OxQzylUaITiOEXHVypQGixwBxLHEedsuc6+ELrjS4aHDR4GIMeGOAA+bODX7qHHiSK7nWGV88tmx8MV9l9eyW+ARXGiE6jRBy1cmVBogeA8SxxHnYLXOuhy+40uCiueE1N4p0AtkB19sBLxIH3LcZccB8bQanovlEruRaNGa4f+diBvlKI0SnEUKuOrnSANFjgDiWOA+7Zc718AVXGlw0uGhwMQa8McABc+cGP3UOPMmVXOuMLx5bNr6Yr7J6dkt8giuNEJ1GCLnq5EoDRI8B4ljiPOyWOdfDF1xpcNHc8JobRTqB7IDr7YAXiQPu24w4YL42g1PRfCJXci0aM9y/czGDfKURotMIIVedXGmA6DFAHEuch90y53r4gisNLhpcNLgYA94Y4IC5c4OfOgee5EqudcYXjy0bX8xXWT27JT7BlUaITiOEXHVypQGixwBxLHEedsuc6+ELrjS4aG54zY0inUB2wPV2wIvEAfdtRhwwX5vBqWg+kSu5Fo0Z7t+5mEG+0gjRaYSQq06uNED0GCCOJc7DbplzPXzBlQYXDS4aXIwBbwxwwNy5wU+dA09yJdc644vHlo0v5qusnt0Sn+BKI0SnEUKuOrnSANFjgDiWOA+7Zc718AVXGlw0N7zmRpFOIDvgejvgReKA+zYjDpivzeBUNJ/IlVyLxgz371zMIF9phOg0QshVJ1caIHoMEMcS52G3zLkevuBKg4sGFw0uxoA3Bjhg7tzgp86BJ7mSa53xxWPLxhfzVVbPbolPcKURotMIIVedXGmA6DFAHEuch90y53r4gisNLpobXnOjSCeQHXC9HfAiccB9mxEHzNdmcCqaT+RKrkVjhvt3LmaQrzRCdBoh5KqTKw0QPQaIY4nzsFvmXA9fcO01d+5cE+rPVohK+UcNGAOMAcYAY4AxwBhgDDAGGAOMAcYAY4AxwBhgDDAGpGLABJxgpHHSpwC56mOKT0Su5KpTAZ2fivlKrjoV0PmpmK/kqlMBnZ+K+UquOhXQ+amQr0FvUeQJQm8g6fxk/9+fivmqkz+5kqtOBXR+KuYruepUQOenYr6Sq04FdH4q5qterv8Fkwq2BwEIK/0AAAAASUVORK5CYII=" } }, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 3. Split the data into train, dev, test sets\n", "\n", "Shuffle ข้อมูล \n", "หาจุดตัดใน data set \n", "\n", "![image.png](attachment:image.png)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train, the_rest = train_test_split(new_data, train_size=0.6)\n", "dev, test = train_test_split(the_rest, test_size=0.5)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1935\n", "645\n", "646\n" ] } ], "source": [ "print(len(train))\n", "print(len(dev))\n", "print(len(test))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 4. Save ใส่ไฟล์" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train.to_csv('train-blog-gender-dataset.csv', index=False)\n", "dev.to_csv('dev-blog-gender-dataset.csv', index=False)\n", "test.to_csv('test-blog-gender-dataset.csv', index=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "loaded_data = pd.read_csv('train-blog-gender-dataset.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "loaded_data.head()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "vscode": { "interpreter": { "hash": "34368ba4908ea1be08ba769dfb7764ab7f8ead2384ebb5604cb86637573696f7" } } }, "nbformat": 4, "nbformat_minor": 4 }