NLP on jeffcarp

用20行Python构建Markov Chain语句生成器

Sat, 04 Jul 2020 10:44:41 -0800

A bot who can write a long letter with ease, cannot write ill.

—Jane Austen, Pride and Prejudice

这篇文章将引导您逐步学习如何使用Python从头开始编写马尔可夫链(Markov Chain)，以生成好像一个真实的人写的英语的全新句子。简·奥斯丁的《傲慢与偏见》(Pride and Prejudice by Jane Austen) 是我们用来构建马尔可夫链的文字。 Colab 上有一篇可运行的笔记本版本。

Read the English version of this post here.

Setup

首先下载“傲慢与偏见”的全文。

# 下载Pride and Prejudice和并切断头.
!curl https://www.gutenberg.org/files/1342/1342-0.txt | tail -n+32 > /content/pride-and-prejudice.txt

# 预览文件.
!head -n 10 /content/pride-and-prejudice.txt

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  707k  100  707k    0     0  1132k      0 --:--:-- --:--:-- --:--:-- 1130k
PRIDE AND PREJUDICE

By Jane Austen



Chapter 1


It is a truth universally acknowledged, that a single man in possession

添加一些必要的导入。

How are Words Represented in Machine Learning?

Sat, 13 Jul 2019 01:31:29 +0000

Note:This post had a good run, but it is now very out of date. Check out newer content around tokenizers for a more up-to-date view.

Machine learning on human languages is a super exciting space right now. Applications are exploding—just think of how many natural language ML models it takes to run a smart assistant, from transforming spoken audio to text, to finding the exact part of a web page that answers your question, to choosing the correct words with the correct grammar to reply to you.

Build a Markov Chain Sentence Generator in 20 lines of Python

Wed, 16 Jan 2019 10:44:41 -0800

A bot who can write a long letter with ease, cannot write ill.

—Jane Austen, Pride and Prejudice

This post walks you through how to write a Markov Chain from scratch with Python in order to generate completely new sentences that resemble English.

The text we’ll be using to build the Markov Chain is Pride and Prejudice by Jane Austen. You can follow along here or grab a runnable notebook version of this post on Colab.