Software Engineering on jeffcarp

Favorite Technical Books

Fri, 29 Nov 2024 00:00:00 +0000

Here’s a quick list of my favorite software engineering books I’ve read over the years. I plan to keep this list updated as I discover (or am recommended, hint hint) more.

Crypto: How the Code Rebels Beat the Government—Saving Privacy in the Digital Age, Levy
Cryptography Engineering: Design Principles and Practical Applications, Ferguson, Schneier, Kohno
Designing Data-Intensive Applications, Kleppmann
Designing Machine Learning Systems, Huyen
Reinforcement Learning, Winder
Hands-On Machine Learning with Scikit-Learn and TensorFlow, Geron
Site Reliability Engineering, Beyer, Jones, Petoff, Murphy
Linux Kernel Development, Love

Quickly saying “no” to ideas is harmful

Mon, 23 Sep 2024 00:00:00 +0000

I’ve observed during my software engineering career the different reactions senior engineers have when new ideas are brought up.

Sometimes when an idea is presented, usually by a junior engineer, a senior engineer will chime in very quickly to emphatically describe why it won’t work. I think this is very harmful to team culture for a couple reasons.

First, it reinforces the bias toward authority for senior engineers. When any engineer brings up an idea, even if it’s overall a bad idea, there is always a nugget of truth – they observed a real issue and offered a real solution. Immediately shooting down ideas discards this truth.

Deciding between a static and dynamic site

Thu, 04 Jan 2024 00:00:00 +0000

Hi, I’m trying to blog more this year. Maybe not every day, but more frequently.

In a recent post I mentioned I’ve been working on a fast frontend for a Chinese-English dictionary. In the spirit of “blog about what you’ve struggled with”, the biggest challenge I’m currently facing is choosing between designing the site as front-end heavy or back-end heavy.

I’ve written two initial versions of the site: one as a fully static website and one as a fully backend app in Go. Despite the server-side version being written in Go, I’m not super happy with the load time of the app. Even when the server is warmed up, the initial document load time is 30-40ms, whereas the static site is 10-20ms.

Considering File Formats for ChineseDict

Tue, 05 Dec 2023 00:00:00 +0000

Recently I’ve been hacking on ChineseDict, a fast web frontend to look up Chinese words. I started the project originally to fulfill a specific use case: when practicing chatting with friends or watching a video, I need search results in milliseconds, not seconds.

The website uses the CC-EDICT dictionary to power the results. I now want to expand each term with examples and extra data in a way that’s easy to store on disk and update via git commit.

How to clone a Google Source Repository in Working Copy iOS

Sat, 19 Dec 2020 02:00:00 -0700

I recently went through this process and couldn’t find a guide (though I swear one existed at some point in the past). Here’s how to clone a git repository from Google Source Repositories in the Working Copy iOS or iPadOS app:

Navigate to https://source.cloud.google.com/
Pick out the repo you want to clone and open the clone dialog (it looks like a “+” icon)
Go to the “Manually generated credentials” tab
Click on “Generate and store your Git credentials”
Go through the authentication flow
You should be prompted to copy a shell command onto your clipboard
Go to the Working Copy iOS app
Go to Settings (the gear icon) > Authentication Cookies
Tap the “+” icon and import from clipboard
You should now be able to clone the repository using the https://source.developers.google.com/p/... URL

Hope that helps if you’re in the same boat.

Grace Hopper 2019 Trip Report

Sun, 16 Aug 2020 00:00:00 +0000

Despite this trip report being over 9 months late, I wanted to share it because I can’t stop thinking about how positive an experience this conference was. Grace Hopper is the largest women in tech conference in the world, with around 25,000 attendees flying into Orlando, FL from all parts of the world for the 2019 conference. In previous years I had been interested in attending but hadn’t gotten the chance—and I (as a man) also strongly did not want to take the spot of a potential woman visiting the conference. This year I was lucky to get the chance to attend and thought it would be a valuable experience to learn how to be a better ally and advance the cause of women in tech, as well as learn about the latest tech from across the industry.

用20行Python构建Markov Chain语句生成器

Sat, 04 Jul 2020 10:44:41 -0800

A bot who can write a long letter with ease, cannot write ill.

—Jane Austen, Pride and Prejudice

这篇文章将引导您逐步学习如何使用Python从头开始编写马尔可夫链(Markov Chain)，以生成好像一个真实的人写的英语的全新句子。简·奥斯丁的《傲慢与偏见》(Pride and Prejudice by Jane Austen) 是我们用来构建马尔可夫链的文字。 Colab 上有一篇可运行的笔记本版本。

Read the English version of this post here.

Setup

首先下载“傲慢与偏见”的全文。

# 下载Pride and Prejudice和并切断头.
!curl https://www.gutenberg.org/files/1342/1342-0.txt | tail -n+32 > /content/pride-and-prejudice.txt

# 预览文件.
!head -n 10 /content/pride-and-prejudice.txt

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  707k  100  707k    0     0  1132k      0 --:--:-- --:--:-- --:--:-- 1130k
PRIDE AND PREJUDICE

By Jane Austen



Chapter 1


It is a truth universally acknowledged, that a single man in possession

添加一些必要的导入。

I'm Joining Waymo

Sun, 10 Nov 2019 15:38:38 -0800

Quick life update: I’ve left the Chrome team and joined Waymo (formerly the Google self-driving car project). I’ll be working on ML infrastructure.

It was a fantastic whirlwind 3 years working on infrastructure for Chromium and helping to–in a very small way–push the open web forward. On the team I launched wpt.fyi, a resource to help align the APIs of all browsers. I worked on syncing source code across repos. I launched a couple TensorFlow ML models. And I helped make the bug tracker quicker and more useful for everyone in the project.

Things I Learned as a First-Time Intern Host

Sat, 31 Aug 2019 11:13:25 -0700

I hosted an intern for the first time this summer. It was my first time being somebody’s manager and it became a huge learning experience for me as well as a really fun time. My intern worked on adding many features to velocity-tracking charts, rewriting our ML models in TensorFlow 2.0, and a few other projects.

Here are the biggest areas where I struggled as a host and the important lessons I took away from those experiences.

How I Host Static Sites With Automatic Deploy on Green

Sun, 03 Feb 2019 23:19:44 +0000

This site, jeffcarp.com, is written in markdown and uses the Hugo static site generator. This post walks you through how I set automatic building, testing, and deployment to Firebase hosting.

Project Setup

I assume we’re starting from a working Hugo project. For more on how to set that up, see the Hugo docs.

Testing Setup

I want the site to be Deploy-on-Green (i.e. only if it passes the tests). The CI setup I use is GCP Cloud Build. I develop and debug these tests locally using the local-cloud-build command documented here. The steps I want are:

Build a Markov Chain Sentence Generator in 20 lines of Python

Wed, 16 Jan 2019 10:44:41 -0800

A bot who can write a long letter with ease, cannot write ill.

—Jane Austen, Pride and Prejudice

This post walks you through how to write a Markov Chain from scratch with Python in order to generate completely new sentences that resemble English.

The text we’ll be using to build the Markov Chain is Pride and Prejudice by Jane Austen. You can follow along here or grab a runnable notebook version of this post on Colab.

It's OK to Make Mistakes in Coding Interviews

Tue, 25 Sep 2018 01:02:51 -0700

Imagine you’re an interviewer in a software engineering interview. The interviewee writes a method and it’s completely wrong. They go on to implement other parts of the problem, then later realize there’s a bug and come back and fix the original method.

One thing that isn’t super clear for both interview candidates and interviewers is how to treat mistakes like this. The candidate definitely made a mistake. But their final implementation is correct. Should they be penalized?

Doing Cryptography in TensorFlow

Sat, 23 Jun 2018 13:00:14 -0700

On the left: the Feistel Network from the DES cipher, implemented below. On the right: a deep neural network.

TensorFlow is a popular machine learning framework. If you look under the hood, TensorFlow is a general platform for doing computation over tensors in the structure of a graph.

Understanding the Security of Cryptographic Hash Functions

Mon, 16 Apr 2018 21:33:26 -0700

Hash functions are an extremely versatile tool that you can find nearly everywhere in software engineering.

Bitcoin mining is done by repeatedly computing SHA256 hashes.
git stores files internally using the SHA-1 hash of the content as the filename (content-based addressing).
When you download an app, your phone verifies that the app you’re about to run is the app you intended to download by computing a checksum (a hash of the entire app) and comparing it to one provided by the app store. If the values are different, that means someone might have changed the contents of the app.

This article intends to give you a deeper understanding of cryptographic hash functions, when to use them, and how to think about their security properties.

Thoughts on Stanford's CS255 Intro to Cryptography

Sat, 31 Mar 2018 12:31:22 -0700

I’ve always been interested in Cryptography – the field is a really compelling marriage of beautiful mathematics with critical real-world applications. Much of the modern internet would not be possible without the clever math behind asymmetric encryption.

This Winter quarter I took Stanford CS255: Introduction to Cryptography. To be honest, it was quite challenging as I was still working full time as a software engineer on Chrome, and I was never that strong in the math department. However, I got a ton out of this course and solidly recommend it to anyone interested. The professor – Dan Boneh – was amazing and made the complex topics approachable. He publishes the textbook for the course for free here. Here are some interesting things I picked up from the course that I want to take with me.

How to Export Evaluation Results in Tensorflow

Fri, 05 Jan 2018 21:33:26 -0700

2019 update: just a heads up, this post is about TensorFlow 1.x. For TensorFlow 2.x, you probably want to check out Keras custom callbacks.

In TensorFlow if you’re using a tf.estimator model, for instance tf.estimator.DNNLinearCombinedClassifier, and as part of your automated training infrastructure you want to save the evaluation results as a JSON file, it’s not super straightforward, so here’s how to do it.

Let’s say you define your EvalSpec like this:

eval_spec = tf.estimator.EvalSpec(eval_input_fn,
  steps=hparams.eval_steps,
  exporters=[exporter],
  name='eval')

You’ll need to write a new exporter class that will take the eval_result from your evaluation step and save it to a file using the GFile API.

Example: Save and Load a TensorFlow Model

Sun, 19 Nov 2017 19:07:30 +0000

2020 update: just a heads up, this post is about TensorFlow 1.x. For TensorFlow 2.x, you probably want to check out this guide .

This post details how to save and load a TensorFlow model using the DNNClassifier API.

The key idea here is that you define a function or a class beforehand that takes a model directory (in which it will save and restore the model parameters), adds that to RunConfig, and returns a tf.contrib.learn.Estimator, for example, tf.contrib.learn.DNNClassifier. See make_estimator for more details.

Chrome Security Architecture

Sun, 22 Oct 2017 19:07:30 +0000

While in Tokyo for BlinkOn8, a gathering for Blink and Chromium contributors, I gave a talk at a coding school about the security architecture of Chromium. Here are the slides.

I’m not an authority when it comes to security or specifically security in Chrome, but I’m really interested in it and I hope this deck has some information you find interesting as well.

Where Web Payments are Going

Sun, 31 Jul 2016 15:15:40 +0000

I gave a talk at the SF Payments Engineers meetup about Web Payments. Here’s the blogified version of that talk. Check out the slides from my talk here.

Mobile checkouts are a terrible experience for customers. Both consumers and merchants are both feeling the pain. Mobile checkouts convert about 66% worse than desktop checkouts.

A slide from the PaymentRequest presentation at Google I/O 2016.

Part of the problem is that every company needs to build their own credit card form. Checkouts range from great UX to terrible UX, but the heterogeneity is what really makes it a pain for consumers.

You are an engineering manager whether you realize it or not

Mon, 25 Jul 2016 21:33:26 -0700

Over the past few months I’ve mentioned to friends that I want to learn engineering leadership skills. Each time the reaction is: “you want to get into management??” That’s not how I see it.

As engineers we hold the lone wolf in high regard, the hoodie-wearing coder hacking away in the corner at a genius project. But large projects that deliver a ton of value are made by teams of engineers, not lone wolves.

Reading Papers: Bufferbloat, SSL Warnings, Orleans, and more

Tue, 24 Mar 2015 00:00:00 +0000

Inspired by the article Should I read papers? by Michael R. Bernstein and the amazing Papers We Love GitHub repository, I set out at the beginning of January 2015 with the goal of reading 15 academic papers. In this post I want to share with you my notes for the 4 I loved the most.

Takeaways

In a certain way reading papers is a comforting activity. In a world where the front-end framework you’re using might be eclipsed in usefulness in 1 or 2 years, many of these papers present ideas that are as applicable today as they were 43 years ago. Here are the publish dates of the 15 papers I read:

Visualizing JavaScript Project Structure

Mon, 12 Jan 2015 00:00:00 +0000

Recently I thought to myself: fairly frequently I dive into a big JS codebase and need to poke around for a while to get acquainted with the project structure, wouldn’t it be nice if I could have a more visual way of seeing the whole thing to make sense of it all?

It was this need I was trying to fill when I embarked on my open dev project last Friday. In this post I’ll try to walk through the steps I took so you can replicate them. If you want to just to play with the final project, it’s available as an npm module named jsviz (npm: jsviz, GitHub: jeffcarp/jsviz).

How System Calls Work

Thu, 18 Sep 2014 16:36:39 +0000

Why should you care about syscalls?

As a web developer, learning about syscalls and the infrastructure around them can make you feel quite a bit more confident in debugging and reasoning about how systems will perform. Ruby and C++ both have their own idiomatic ways of opening files, but in the end they both end up using the syscall open(). This is because userland processes (like web applications) have only one way of communicating with the operating system: syscalls.

Weird and Cool things from Linux Kernel Development

Fri, 12 Sep 2014 00:06:15 -0700

Recently I picked up Linux Kernel Development by Robert Love since I like his posts on Quora. Here are the things that I thought were cool or just surprised me.

Threaded Trees

Are cool. They act both like trees and linked lists.

Child processes vs Threads

The only difference between a child process and a thread in Linux is whether the CLONE_VM flag was passed to clone(), which copies the parent’s address space which allows the child to access the parent’s memory.

Unforseen Perks of Pair Programming

Sun, 15 Jun 2014 19:03:46 +0000

As someone who had never pair programmed before, it was exciting to get thrown into the deep end during my first week at Braintree where engineers pair nearly 100% of the time.

The relative merits of pair programming have already been spoken about at length.^[1]^[2] This post is not an attempt to argue one way or another. Whether it works for any organization is probably too context-dependent for any axioms I could lay down.

Interviewing 2 Years in: What Worked

Fri, 06 Jun 2014 00:00:00 +0000

I’m almost pee-my-pants excited to announce that next week I’ll be joining the team at Braintree and Venmo to work on their JavaScript SDK.

Now that my job search is over, I want to share the lessons I learned from interviewing as a software engineer 2 years into my career. Most of these are not new (or specific to being 2 years into your career), but these points are what specifically helped me the most.

Intro to Angular.js Talk

Mon, 28 Apr 2014 23:40:24 +0000

On April 24 I gave a talk, Introduction to Angular.js, to the SF HTML5 Live Code meetup group. Here’s the screencast:

Things learned while preparing for Angular Live Code

Tue, 22 Apr 2014 00:00:00 +0000

This week I learned that there’s nothing quite like signing up to teach something to make you realize you don’t actually know the material all that well.

I’m giving an introduction to AngularJS talk this Thursday at Startup House SF. You should come by if you want to learn about AngularJS.

It’s Tuesday and 124 people have signed up and I’m kind of nervous to say the least. I’ve tried to channel my nervousness into preparation (we’ll see if it pays off or not). Here are a few things I’ve learned in the process.

Two interesting IE JavaScript quirks

Fri, 14 Mar 2014 00:00:00 +0000

In the past month I’ve diagnosed and fixed a couple particularly pernicious bugs relating to JavaScript quirks in Internet Explorer. Hopefully if you haven’t run into these before, this will save you some pain in the future.

1. `window` does not inherit from `Object.prototype`

Oddly enough, according to the ECMAScript specification, the global window object does not necessarily have to inherit from Object.prototype. In Chrome, FF, and Safari, it does, but in IE it does not.

JavaScript's Mutative vs. Non-Mutative Array Methods

Thu, 20 Feb 2014 12:00:04 +0000

This is an issue that has endlessly tripped me up in JavaScript and I hope this post will help clarify things a bit and hopefully discover a pattern one can use to discern between mutative and non-mutative array methods.

First, let me define what I mean when I say mutative:

// If a method is mutative, that means it changes the original array.
var foo = [1, 2, 3];
foo.reverse();
console.log(foo); // [3, 2, 1]

// Notice how we didn't assign foo to new variable, reverse() acted upon it in place.
// In contrast, non-mutative array methods return a new copy of the original array.

// Here, the new array concat() generates is lost since we didn't assign it
var bar = [1, 2, 3];
bar.concat([4, 5]);
console.log(bar); // [1, 2, 3]

// Now if we assign it, bar becomes the new array.
bar = bar.concat([4, 5]);
console.log(bar); // [1, 2, 3, 4, 5]

Mutative	Non-Mutative
splice	slice
shift, pop	concat
unshift, push	every *
reverse	filter *
sort	forEach *
	indexOf
	lastIndexOf
	join
	map *
	reduce *
	reduceRight *
	some *
	toLocaleString
	toString

NB: the functional methods marked with a * can be mutative if the function you pass to them modifies the original array. In my expierience this has been especially true in the case of Array.prototype.forEach.

Japanese Programming

Mon, 25 Feb 2013 00:00:00 +0000

I’m certainly guilty of being a little to anglo-centric from time to time (pizza is an American food, right?), so I was pleasantly surprised when I found out that you can program in languages other than English. You can see the full list on Wikipedia. Since I’m into both computers and Japan, I thought I’d give a few of them a spin.

Dolittle ドリトル

According to its Wikipedia page, Dolittle is a programming language for education developed by Osaka Electro-Communication University. Its creators called it Dolittle because they wanted to make a programming language that made it very easy to start programming.