Off by one

Div III: Empowerment software

2013-05-01T00:00:00Z

The following is a summary of my 2013 senior thesis at Hampshire college. Many of the themes I identified at the time still resonate with me, so I decided to include it in my blog. Some of it I roll my eyes at, but that's OK.

Introduction

Picking a Div III

My Div III started twelve months ago. I was looking for a large-scale project to create or join. I was considering a lab-based biology research project. But, I felt that that sort of project wouldn't play to my strengths. I was attending a Biomath lecture series at Smith College. The purpose was to present the complex intersection of computational techniques and life sciences. It was also intended to encourage young people to work with computational and biological fields. At the time, I was TA-ing Organic Chemistry (it had been my third semester of chemistry in a row). I didn't want to be a chemist. I wanted to be a doctor, and I had to study chemistry to go to medical school. Nonetheless, chemistry was interesting, and it was becoming less intimidating than it had been the year before.

I heard a lecture on Genetic Programming by Lee Spector. The talk was fascinating. He described how one could create a computer system where programs evolved to solve problems. It had been my first formal introduction to GP. During the talk, I remembered a conversation I had had with my friend Omri Bernstein. He was telling me about how he thought there was probably an interesting project that would combine Genetic Programming and Organic Chemistry. In the QA of Lee's talk, I raised my hand, "Has anyone ever combined GP and organic chemistry." He said that he hadn't heard of anyone doing that, but suggested that I meet with him and talk about it.

I considered doing a Div III in that subject area. GP and chemistry seemed to leverage my strength in computer programming in ways that an ambiguous biology project wouldn't.

I met with Lee a few times, once with a Smith chemist, David Gorin. Lee described GP to Dave. Dave thought of a project that might work. I was seriously considering doing the project.

Then, the biomath lecturer coordinator announced that students could apply for an NSF-funded Biomathmatics fellowship. I decided to apply with this project. If I was accepted, I would make this my Div III. I was, and I did... at least at first.

The Summer

Before the Biomath fellowship, I had arranged an internship at the Center for Resuscitation Science at Beth Israel Deaconess Medical Center in Boston. Now that I had the fellowship, I would have to juggle both. But, I didn't yet know what my responsibilities at BIDMC were. Lee and the fellowship coordinator agreed that it would be OK if I were to travel to the 5 college area weekly for meetings. Luckily for me, the job at BIDMC was pretty rote. I had to do a lot of manual data entry. I had no ongoing projects. They were fine with me working 3 days per week. I worked Monday, Tuesday, and Wednesday. On Thursday I would take the bus to Hampshire and attend a meeting with Lee and his research students. I stayed on a friend's couch and bussed back the next day.

With Lee's group, my first task was to build the data infrastructure that would allow for the chemistry GP project. I was dealing with data from all sorts of different sources, building a database that would be easily accessed by GP. This project is described in the ChemGP section. At the same time, I was doing very repetitive work copying data from one computer system to another in the hospital. I realized that I had the technical tools to improve the BIDMC computer systems used for research. But, no one at BIDMC was willing to entertain the idea. In August, I wrote an essay about my experiences and frustrations with clinical research. After many iterations, this piece will be published in The Scientist in May.

The fall

In the fall, I was working on my Div III while doing three advanced learning activities. I was teaching Intro to Web Development, TAing Organic Chemistry II, and TAing Genetic Programming. The largest commitment by far was teaching Web Development. I planned the classes, reviewed student work, and met with students. TAing Orgo was less work, but still took time. I had weekly TA sessions, went to class, reviewed material, and met with struggling students for extra help. TAing Genetic Programming was my smallest commitment. I went to class and occasionally met with students who needed help.

By the end of October, I had made some progress on my Div III project. I had built WebGP (described below) and interfaced ChemGP with it to do runs. It was around this time that I went to a Hackathon hosted by Communicate Health at UMass. My team produced CarrotStick (described below). We won the Hackathon. That project started to change my priorities. Building software that related to people was very exciting.

A few weeks later, the team went to another Hackathon. This one was hosted by Mass General Hospital and MIT's Hacking Medicine group. The subject was Global Health. We built Trext (described below) and received a second-place prize. Trext became the largest part of my Div III. I worked on Trext full time (+ 50%) for several months.

As the technical demands of Trext plateaued (while I waited for the business aspects caught up), I started working on Modfinder, a tool to help Hampshire students find housemates. The impetus behind the project was primarily to learn a new technology stack. The project is described below.

Throughout my Div III, I collaborated with different individuals. On ChemGP, I worked with Omri Bernstein, especially in the beginning. The opinion essay I wrote was co-authored with Dr. Martina Steurer, MD, although I did the primary writing. On CarrotStick, I collaborated a lot with Kira McCoy, and somewhat with James Matheson and Amit Ringel. I worked on Modfinder with Seth Toles.

My preferred technology stack for web apps has changed significantly over the course of the year, mostly on the server side. I started the year working primarily with PHP/MySQL. Now I prefer Node.js, MySQL, and MongoDB.

webGP

Go to Github project
~Demo~

webGP is a web-based, javascript tool that allows anyone to do symbolic regression GP runs.

What is GP

Genetic programming is a technique that allows computers to write programs. The genetic programmer must first write a function that can evaluate program-candidates. This is called the fitness function. This function evaluates a candidate program and returns some form of score. The genetic programmer must also build a random code generator that can produce random programs. With these two components, one can build the first steps of a GP system by generating many random programs and evaluating them with the fitness function. Typically, none of the random programs come close to a good score, but some are better than others. Thus far, this is mildly interesting, but not powerful. In order to leverage principles of Darwinian evolution, programs must be able to "reproduce."

A simple genetic programming system is completed when programs can mutate and/or breed with one another. A mutate function takes a program and returns a modified version of that program. A crossover function takes two programs, removes a piece of one, and replaces it with a piece of the other. To run the finished GP system, generate a series of random programs. Rate their fitnesses. Pick some of the better programs and breed them to produce a new series. Each series is called a generation. If the system is working properly, the programs' fitness should improve over time.

Genetic programming is a deep and rich field. There are countless papers published on variations of each of the concepts explained above.

What is Symbolic Regression

Symbolic regression is a GP technique that attempts mathematically explain data. In some ways, it is analogous to linear regression. In linear regression, two-dimensional (x, y) data is modeled by a linear function. Symbolic regression is similar except there are no requirements on the type of function and the data can be n-dimensional (x, y, z, ...). Functions are generated through GP. Symbolic regression is powerful when searching through complex data.

Motivation

As scientific tests become less expensive and more abundant, scientists are facing increasingly large datasets (source). Symbolic regression has potential as a tool for scientists to make sense of the mess they're in. But, it faces a problem. The barrier to entry is too high. This was the inspiration for building webGP. I wanted to give people a way to run GP on data they have sitting in an excel file or database.

Function choice and GP parameters

Function choices and GP parameters. I made an interface that allows the user to decide what mathematical operations the evolved programs may employ. The user can also specify parameters about the run (its length, size of generations, etc).

Visualization of results

Real time
- Visualizing the best error in each generation as the data comes in
- Visualizing the best program's size as it is generated
After the run
- Best program's predictions against fitness cases. A graph that plots two series. The fitness cases and the best program's prediction of what they should be. This gives the user a rough understanding of how good the best program is.
- Piano plot. This graph shows each individual program's score plotted on the y-axis and its (arbitrary) ID on the x-axis. The IDs are based on score and generation. Each point is shaded based on its length. This plot can give the user a sense of the diversity of the population in each generation
- Size vs error vs age. This graph shows how the relationship between size of programs, their error, and their generation. It gives the user a sense of how size and error changed over generation time.

Process of building

At first, I started building a simple GP engine in javascript as a way of learning GP. I had been working on other projects in Clojure, but didn't have a good feel of the language. I didn't want to let this technical clumsiness inhibit my learning of GP. Once I had the engine working, I realized how easy it would be to make a web front-end. First, I just made the data table and GP options. Then, I wanted to see how the process was going in real time. I started learning D3.js, a graphing library. I made some classes for bar charts and scatterplots--the two types of visualization I eventually wanted. With those classes, I made some powerful graphs.

Challenges

When webGP is used for small problems, it's great. Chrome's V8 javascript engine is lightning fast, and webGP is written well enough to harness its speed. But, because I wrote webGP to store and analyze every program ever produced by GP, it eats up a lot of RAM. If you have many generations, and each one is very large, it crashes the browser tab. Also, some of the visualization techniques performed after the run need to output a point for every program. This can take some time. I did make a modification to have the scatterplots only reflect a representative sample of the population. This improved speed considerably.

What would come next

To mitigate the memory problem, I would send the programs to a node.js server to be stored. Node.js has the capacity for real-time client-server interaction with the use of websockets. This means once programs are created and evaluated, they can be stored elsewhere. This frees the client to run as long as it needs to. It also opens up the door for multiple computers all running the same problem.

Status

This project is open source and on GitHub.

ChemGP

In this project, I sought to find mathematical relationships between several attributes of molecules and their reaction rate constant in a cycloaddition reaction. I used symbolic regression GP in the search. Cycloadditions are reactions that make a new hydrocarbon ring (or "cycle"). They are synthetically powerful for many reasons, but here I will focus on their utility in "bioorthogonal" chemistry (BC). BC is the study of synthetic chemical reactions that run inside an organism but do not interfere with the endogenous biochemical reactions. Applications of BC include attaching fluorescent molecules to proteins without the standard transgenic mutations required for the technique. These "tagging" reactions could be used by biologists or clinicians for diagnostic purposes. Bioorthogonal chemistry might one day be used for therapeutic purposes.

In order to be useful, BCs must adhere to a rather stringent set of requirements. They must react selectively (have little to no side reactions). They must not be cytotoxic (deadly to the tissue or organism). They must be kinetically accessible (they must happen quickly enough and in high enough yield to be measured). Many of the synthetic techniques used by organic chemists require there to be no water in the reaction. BC reactions must run in water. Organic chemists can control the temperature, pressure, and pH of their reactions, BC reactions must be under physiological conditions. These criteria make BC reactants hard to find.

In this project, I used symbolic regression GP to search for a mathematical relationship between known properties of BC cycloadditions and their reaction rate constants (k). In order to start with GP, I had to build a significant amount of cheminformatic infrastructure. Essentially, I needed a database of reactions. I needed many parameters about each molecule in the reaction--two reactants, a solvent, and a product. I also needed information about the reaction (its pressure, temperature, reaction rate).

I built the database with PHP/MySQL and javascript. I employed the Chemdoodle web framework, an HTML 5/javascript chemistry toolkit. They have an API service which was purchased for chemical computations. Using Chemdoodle, I manually drew dozens of reactions from two papers and entered the reaction parameters. Chemdoodle's API calculated parameters for each molecule and I wrote code to save these calculations into MySQL.

In order to get the data in a format that allowed webGP to be used, I needed to write code to translate the multidimensional MySQL data into a linear list. Each reaction consisted of four molecule rows, one reaction row, and several rows to indicate the relationship. Each row had up to 16 fields. I had to translate this information into a simple list. To do this I prefixed all molecule data with the role it was playing in the reaction (diene-molecular weight, dieneophile molecular weight, solvent molecular weight, etc). From this, I was able to create a simple table representing all the data. From this I used webGP to generate functions that took every reaction variable and generated the reaction rate constant.

I found that the webGP engine that had worked well for simple problems was not effective with this problem. The programs evolved but plateaued early in the run (they did not continue to improve). In order to improve the GP performance, I patched webGP to include "trivial geography" in the mating functions. This technique selects the most fit programs from each area in the population, not the population at large. This allows for slightly less fit programs to survive longer and preserves diversity in the population.

The implementation of trivial geography made the runs' plateaus come later, and their fitnesses improve.

I also needed to modify the code to include separate data categories for training and validation of the programs. Certain reactions were randomly selected to train (be used in the fitness evaluation) while others were left to evaluate the fitness of the best programs from the run. Because these data were not used in the training, their incorporation reduces the degree to which the evolved programs are biased towards the dataset.

I presented this project at a 4CBC event in November 2012. The slides from this presentation are available here.

Status: This project is open source and on GitHub.

CarrotStick

Carrot Stick is a piece of web software written at a hackathon hosted by Communicate Health last fall. We were tasked with writing software that helped communicate public health messages. A team of four (myself, James Matheson, Kira McCoy, and Amit Ringel) worked for two days to produce Carrot Stick.

The abstract from Kira McCoy's Div III:

Smoking is the leading cause of preventable death in the US (Danaei, Ding et al. 2009), and accounts for 1 in 10 deaths worldwide (Park, Tudiver et al. 2012). Information is not enough to change behavior. Many smoking cessation interventions have harnessed the power of social relationships, however none have provided direct messages in loved ones' voices. The literature shows that "The person-to-person spread of smoking cessation appears to have been a factor in the population level decline in smoking seen in recent decades" (Christakis and Fowler 2008). People are connected, as is their health. Social relationships and expanded agency can be used to help people achieve health goals like smoking cessation. Services such as counseling may have short-term impacts on smoking habits, but concern from a loved one may have lasting effects.

A feasibility study of the program "CarrotStick" was conducted to assess a novel smoking cessation intervention. CarrotStick empowers smokers to quit by providing unique support from family and friends. A smoker signs up online, invites friends to record messages and when she feels the urge, calls a number and hears either a "carrot message" (positive reinforcement) or a "stick" (tough love). Participants were smokers over the age of 18, who have the intention to quit. They were recruited through posters and online advertisements, leading them to the study website. Participants completed three surveys throughout the intervention, assessing smoking behavior, satisfaction with CarrotStick and other feasibility items.

The results being analyzed include assessing the number of recorded messages, types of messages, frequency of listening, attrition rates, impacts of reminders, software satisfaction, and limited smoking cessation efficacy.

This study will provide insight into the impact and feasibility of implementing CarrotStick on a larger scale.

Carrot Stick was unanimously chosen by the judges to receive the top prize. Our group was elated. We were surprised by the degree to which our collaboration had improved our final product. We were all surprised by how good Carrot Stick looked after just two days.

Kira decided to keep working. She changed her Div III. She wanted to run a feasibility study on real smokers to see if Carrot Stick really helped people. To do this, I had to do some ongoing work. I built out different views for the app for participants in the control group and participants in the treatment group. I also designed and developed a survey form that combined three tobacco dependence measurements that Kira found. Finally, after noticing that people forgot to call into Carrot Stick, I made an SMS reminder system.

Carrot Stick was built with PHP, MySQL, the Twilio voice API, jQuery, and underscore.js. It was deployed on the PaaS Heroku. It was the first time I deployed on a PaaS.

Status: The status of this project is unclear. We may continue development with grant money in the fall. If we don't, we will open-source the project.

Trext

After our group's success in the first Hackathon, we decided to try again. We heard about Hacking Medicine, an event at Massachusetts General Hospital hosted by the Massachusetts Institute of Technology. The event's theme was global health. They were looking for groups to build software or hardware that could have an impact globally.

Before the event, we brainstormed for our idea. On our mind was the concept of the clinical checklist as presented by Dr. Atul Gawande, author of The Checklist Manifesto.

Dr. Gawande, a surgeon at Brigham and Women's hospital describes how two-minute checklists before certain procedures and operations can cut complications and death by double-digit percentages. As medicine's complexity has increased so have routine errors. Even experts with tens of thousands of hours of training make simple mistakes when confronted with complex, time-sensitive choices. Checklists can ensure that routine precautions are always observed in clinical settings.

Also on our minds was the limited technology in many low and middle-income countries. Building a web app for checklists was tempting, but our target populations don't have reliable or fast internet. However, some technology has penetrated nearly the entire planet. Most people have access to cell phones with SMS, for example.

We thought about building an SMS checklist builder. It would be great for data collection and updating checklists in a way that paper wouldn't. But, SMS is cumbersome, especially on older phones. One of Dr. Gawande's two-minute checklists might take 4 or 6 minutes by text. More importantly, some of the questions weren't relevant.

We stumbled on the idea of using the format of a decision tree to build these checklists. Decision trees can achieve complexity with a short path by eliminating irrelevant questions. We envisioned a simple GUI that would allow basic computer users to build SMS text apps.

At the event, we used a javascript library called JSPlumb as the backbone of our interface. On the server side, we used PHP and MySQL. The SMS functionality was achieved with the Twilio SMS API. It was an ambitious project for a two-day competition. But, 30 minutes before our presentation, we had a working prototype. For the presentation, we represented a clinical protocol for resuscitating infants. Our presentation could only be three minutes long so everything had to be condensed. We won one of the 2nd place prizes at the competition.

The software 2 days after the competition:

There was a palpable feeling that we were really good at this. We wanted to keep going. We thought the texting app was fantastic and we wanted to see it used on the ground in global health settings. But, everyone we showed it to had different ideas of use cases. Most of them were not in the field of global health. We realized we had built a useful design tool.

We thought about using the tool we'd created as a commercial product. But, before we could, we needed to rewrite the entire codebase. The hackathon code wasn't thought out very well and couldn't support new features. We wanted to use Node.js on the server side because of some of its advantages over PHP. But, node is hard to use. We had a lot of interest in the software, and we didn't have time to learn an entirely new technology stack. So, we took on some technical debt. We did a rewrite with the same technology stack. Much of the rewrite happened over winter break while I was shopping the idea around to VCs, talking to lawyers, and pitching to San Francisco-based leads.

The team worked intensely over January. We worked on a batch texting, or "Phonebook" feature in which trees could be started by our clients not the end-users. But, one client wanted to be able to reference information about the end-user in trees. They wanted to say something like, "Hey Zeke, we haven't heard from you in a while, how's the [insert x y or z] going?" We thought about creating certain fields that we could know about each user (name, email, etc), but realized that our different client bases wanted radically different things. Creating rigid fields wouldn't be consistent with what we'd already built: a design tool. We decided that any contact should be able to have n fields associated with this. We had been using the standard relational database: MySQL. We decided to employ a NoSQL environment without strict schemas.

But, we were slightly stuck. We didn't know how to graphically display very abstract, open-ended data. The solution came when we realized how our users would actually import contact data: an excel file. Our users have existing lists of the people they want to text, we just need to give them a way to upload them and view that excel file online. Here's a snapshot of how the Phonebook works with the tree builder.

In February, we refactored the code three times. The third refactor was separating the code that ran trees on phones with the code base that built trees. They are truly different programs and it made sense to run them in different environments so they could be scaled appropriately. (We also realize that the phonebook can and should be separated. That way we'll be able to expose it as a commercial API someday.) We also rewrote the way conversations were stored. We realized that storing them in MySQL required a lot of processing every time a tree log was rendered. We started storing the logs in MongoDB and cut processing time significantly. In this time we also designed a marketing site.

Here is a screenshot of the Trext builder today:

We are currently working on rewriting the Trext SMS program in Node.js. The program that sends and receives SMSs is a perfect candidate for Node. Its IO intensive and would benefit from persistent connections to the database. Additionally, we may want to use websockets in the future to give real-time feedback about tree performance.

The TrextNode project is fully functional, but less tested than the PHP version. After it has been tested and transitioned into use, we will rewrite the backend of the tree builder and web app in Node for a consistent technology stack.

Future of Trext

Trext is going to be a company. We have three investors that are going to support us for a time, and we hope to get enough revenue to be sustainable soon.

Trext also has a future in global health and service. But, it is going to take some very different development. I don't want to get into business models, mostly because I'm not a business person, but I know enough to say that the Trext service isn't affordable enough for global health organizations. We have fixed material costs somewhere between 1.5 and 2 cents per text message.

Additionally, our text messages must come from the United States. This price and restriction makes no sense for a rural clinic in Haiti or Rwanda. However, I'm working on a solution to these problems.

USB GSM Modems (that are AT command compatible) can send and receive text messages from computers. Node.js can send and receive data from USB ports. Theoretically, a computer could be configured as a trext server that does the texting internally. Technically, it wouldn't even need an internet connection, just cell service. If the computer were a laptop, reliable electricity wouldn't be necessary.

To prototype the concept, I purchased a Samsung/Google Chromebook. I rigged it to run Ubuntu Linux from a standard SD camera card. I then configured the system with Git, Node.js, MySQL, and MongoDB. This process took a whole week, which was surprising. At every point in the configuration, I faced new and unexpected problems. I had never compiled C source code for example. I had cryptic errors all the time. When I ran the 'make' command, I got "error (2)." Google was surprisingly unhelpful. Eventually, I found that the problem had been in the system's locale settings. Throughout the process, I got a lot better at manual system configuration and learned that package managers are my friend.

Once I had the baseline system configured to my liking, I learned how to make a disk image with the dd command. I can now replicate the system onto new camera cards for new Chromebooks. The system is scalable. Its total cost is about $350. It runs Trext but could also work as a local web server for other services.

I'm currently working on adapting Trext to run on the system. I haven't yet purchased the GSM modem or begun that side of development, mostly due to cash flow problems. But I expect to be able to make the purchase next week.

After graduation, I have plans to take two prototypes to Haiti and meet my friend Jean. Jean works for OurSoil. They want to test our system. They will use it for following up on trainings they run, sending out surveys, reminders, and educational information. If the test run goes well, we could easily replicate these systems and send them elsewhere.

Status: Code is owned by Trext, Inc.

Modfinder

View Pretty Source

In February I went to EcoHackSF with Seth Toles and others. We built a cool toy, but nothing to showcase here. While we were there we talked a lot about web technologies. We both felt that we took shortcuts frequently while building websites. We wanted to "do it right" for once. We picked the most talked-about new web platform, Node.js. We decided to do a project in Node to learn it. After some brainstorming we decided to build a tool to help Hampshire students find mod-mates. We decided to model our service off the popular dating service OK Cupid. OK Cupid works by letting users ask questions to the entire community and answer others' questions. Users can answer the questions and also express the answer they accept in a partner. We outlined a plan to develop a similar platform. But, our platform would focus more on groups than on pairs. This is because group dynamics are much more complex than one-to-one relationships. You can think about an eight-person mod as a complete graph with eight nodes and 28 edges. There are 28 separate relationships in that mod. We wanted to do all we could to give people a good indicator about what they disagreed on.

First, to see if there was interest, we put up a simple site explaining the idea. In 24 hours we had signups from 20% of the population that participates in the housing lottery. We considered this a success and built the service over spring break.

Since the site's beta launch over spring break, we've had ~234 users sign up, ~7000 questions answered, ~200 questions asked, and ~130 "favorites." We've made ~11,000 match calculations. All of these numbers are growing daily and will continue to do so until the housing lottery.

We've discovered that running a node app takes a lot of hands-on management. It frequently crashes and needs to be restarted. We've been able to do a lot of work with catching exceptions which has prevented the worst of it. We've also realized how immature the platform is. The PaaS that is built in node often has its own failures. We've had pretty bad uptime, though it's gotten better recently.

I'm particularly proud of our matching algorithm which I will include here. I wrote the first pseudocode version, Seth rewrote it in proper Node.js. Then, I wrote the database integration.

var	_ = require('underscore')
,	mysql = require('mysql')
,	connection = mysql.createConnection({
	host:'[omitted]',
	port:[omitted],
	user:'[omitted]',
	password:'[omitted]',
	database:'[omitted]',
	multipleStatements:'true'
});

var numMatches = 0;
connection.query("SELECT * FROM `user` WHERE `numResponses` > 5; SELECT * FROM `questionresponse`; SELECT * FROM `acceptableresponse`;",function(err,results){
	//connection.destroy();
	if(err){
		console.log(err);
		//connection.destroy();
		return false;
	}

	process.on("exit",function() {
		connection.destroy();
	});

	var users = results[0];
	_.each(users,function(user){
		user.responses = _.where(results[1], {userID:user.userID});
		user.acceptable = _.where(results[2], {userID:user.userID});
	});
	_.each(users,function(i) {
		var userMatches = 0;
		console.log("-----------",i.name,"----------");
		_.each(users,function(j) {
			if(i.userID === j.userID)
				return false;
			var overlap = _.intersection(
							_.pluck(i.responses, "questionID"),
							_.pluck(j.responses, "questionID")
						);

			if(overlap.length > 6) {
				var points = 0;
				var possible = 0;
				_.each(overlap,function(questionID) {
					var importance = _.where(i.responses, {questionID:questionID})[0].importance;
					possible += importance;
					if(_.contains(
						_.pluck(_.where(i.acceptable, {questionID:questionID}), "answerID"),
						_.where(j.responses, {questionID:questionID})[0].answerID)) {
						points += importance;
					} else {
						points -= importance;
					}
				});
				if(possible !== 0){
					userMatches++;
					numMatches++;
					connection.query(
						"DELETE FROM `match` WHERE `userID` = ? AND `mUserID` = ? LIMIT 1; \
						INSERT LOW_PRIORITY INTO `match` (`userID`,`mUserID`,`match`,`base`) VALUES (?,?,?,?);",
						[
							i.userID, j.userID,
							i.userID,j.userID,(points + possible) / (possible * 2),overlap.length
						],
						console.log
					);
					var match =  parseInt(((points + possible) / (possible * 2)) * 100);
					console.log("===>",i.name + " : " + j.name + " : " + match + "%, based on " + overlap.length);//possible: " + parseInt(possible) + "/" + overlap.length);
				}
			}
		});
		connection.query("UPDATE LOW_PRIORITY `user` SET `numMatches` = ? WHERE `userID` = ?",[userMatches,i.userID],console.log);
		console.log(userMatches," total matches.");
	});

console.log(numMatches);

Opinion Essay

Published article

Heroic investigations of life and disease have put us on the verge of a health-care revolution. Meanwhile, the software industry is booming; tech startups appear faster than you can say “medical informatics.” But at the intersection of these two complex fields lies dysfunction. Researchers and clinicians, shackled to software that no one in their right mind would use voluntarily, spend countless hours manually copying data from one system to another. And this separation of software development from clinical research is costing lives.

Juggling data

When paper records predominated in medicine, research assistants or medical students manually screened health records for certain diagnoses or laboratory values that would qualify patients for prospective clinical trials. Upon identifying suitable trial candidates and receiving their consent, researchers manually extracted the data of interest, recording the information on yet more paper. Retrospective cohort studies similarly took innumerable hours of searching health records in dusty basements. Data transfer and analyses followed.

Now, the health-care system is in the process of turning away from paper records in favor of digital storage. Research assistants toil no more in dusty basements. Research workflow, however, is stuck in an earlier century. Today's research assistants still use their eyes to screen health records, but on computer monitors rather than paper. Once consent is obtained, relevant information is typically copied by hand from the digital records onto paper, then later recopied into digital research databases.

This is a recipe for bad data. Like the children's game "telephone," in which a group whispers a sentence from one to another, the message can become garbled. In the past, this was unavoidable. Today, there is no excuse.

Time points

One consequence of this disorganization is inconsistency in when key measurements are taken. A hypothetical study might seek to determine if a specific protein level in patients' blood 12 hours after arrival at the hospital is predictive of their outcome. Researchers would compile clinical data at this arbitrary time point, including vital signs, routine clinical laboratory values, and any study-specific measurements. The problem is that none of these data are collected at exactly 12 hours. Routine clinical measurements might be taken only in the morning, for example. Even the study-specific measurements, which the researchers control, may be off for logistical reasons. Such time differences are not routinely noted in the research database, resulting in the lowered statistical power of any conclusions to an unknown degree.

Even worse, important information is often lost: data that are repetitively measured in the clinical setting, like most vital signs, are only transferred to the research database at certain intervals; data measured at in-between times are not saved. This makes it impossible to look at specific trends that could lead to new types of data analysis.

There is a better way

If clinical researchers partnered with computer experts, we could transform the way data are collected, stored, and analyzed. We could reduce human error by mandating fewer manual steps, making research faster and data cleaner. Involving programmers in clinical research could also support more advanced data structures that would allow researchers to record infinite numbers of data points and display them in simple scatter plots to identify optimal time points. And perhaps most importantly, research assistants would be free to spend more time obtaining consent from patients, reviewing literature, and looking for trends in the data, instead of acting as human ethernet cables transferring information between different computer systems and paper records, and back again.

Relying more on computer systems would also make it easier for clinical researchers to find trial participants, by screening for certain lab values in a simple database query. All that is required is commonsense application programming interfaces (APIs), established protocols that allow different software platforms to communicate. In May 2011, realizing the emerging importance of these tools, President Obama directed every federal agency to have an API. Most clinical research teams do not have any APIs, nor the programming expertise to operate and maintain them.

Finally, a more integrated system could also improve treatment. As the deluge of omics data continues, clinical researchers are becoming more and more dependent on computer software to make sense of it all. How can we gain insight from the 3 billion base pairs in the human genome when we can't effectively store and manipulate the 30-odd measures, such as blood pressure and heart rate, that we've had for 50 years?

Writing a short essay is much harder than writing a paper for school. The hardest part for me was that I didn't know what I wanted out of it when I got started. I had something that was bothering me and I thought others should know about. In the summer, I saw that clinical researchers were using computers wrong. This had implications for everyone who's ever been touched by disease. When I was talking to my mom about it, she suggested I write an Op-Ed for the New York Times. My first draft was written with that intention. It was conversational, in the first person. I showed it to a lot of people and got generally positive feedback. People felt I should indeed try to get the message out. There was no consensus on the format.

The New York Times wasn't interested in the piece. I submitted it to some medical journals: nothing. I sent it to a prior mentor: Dr. Ann Zovine. Ann is a physician researcher at UCSF. I worked for her the summer after the second year. She liked the piece and felt that it would be perfect for The Scientist.

I submitted the essay to The Scientist. They didn't get back to me for a long time. School started. I was wrapped up in teaching my web development class and going to hackathons. This paper was on the back burner. Around winter break, they finally got back to me. They liked it, but they needed some credibility behind it. They wanted me to find a coauthor and strip out some of the personal narrative.

I put out feelers and had a few people who were willing to work with me. One stood out. Dr. Martina Steurer-Muller is a pediatrics intensivist fellow at UCSF. She has a lot of experience with research and finds the computer tools totally lacking. She was happy to have her name on the paper.

The editor at The Scientist was pleased that I had found someone. But, we needed to do a complete rewrite. This was really discouraging for me at first. I wasn't sure if the piece had any power without my personal experiences. The first drafts were very weak because I was trying to keep the same skeleton without any of what made it powerful. I didn't have anything good until I started from square one.

Editing and revising this new paper with Martina and the editor took months. First, they were going to publish it only online in the April version. But then someone else at the magazine saw the piece, liked it, and decided it should be published in print in the May issue.

The most surprising thing about this process for me was how long it took. I had this idea nearly a year ago and people are just going to start reading it now. It certainly makes blogging look attractive. I think the process produced a strong paper, but I think there were certain positive qualities lost from the first version of the paper. My takeaway from this is to choose the medium of publishing carefully.

Advocacy

An Example of Advocacy

I've decided to include an email conversation I've had with my high school Biology teacher.

Zeke:

Can I ask your thoughts on an idea I've been incubating? It's not something I'm likely to do now, but it has to do with teaching kids to code and I think you might have some interesting reality checks for me.

Ben:

Sure, ask away. I am happy to play the role of reality checker.

Zeke:

Problem #1: Good web development is too expensive. Programmers are in high demand and you have to pay top dollar to get a great-looking/working site/app. Most cannot afford this, so they use simple tools without much power.

Problem #2: We as a nation (planet?) aren't training programmers fast enough. This is one of the causes for problem #1. Programming can be taught effectively to children. It just isn't. In fact, teaching programming to kids has been anecdotally associated with higher test scores in math and science. Sometimes kids get so good at coding that they are ready for professional work in high school.

Solution to both: Let's think about a company. This company would make most of its money by doing web development contracts. We would bid 30% lower than the market, ensuring more reliable employment. We might even be in the position to pick and choose our clients. This part of the company would produce significant revenue because we'd take on more projects than we could reasonably handle (hang on, I'll explain how this could ever work in a second).

The company would also interact with kids in middle school and high school. We'd do development tutoring in very small groups charging hourly. This aspect of the business wouldn't be a cash cow either, but it would pay for itself. Kids who found they liked coding could go to an intensive development bootcamp over the summer. After the bootcamp, they would be eligible for apprenticeships. Apprentices would be paid $10 / hour and work a lot. These kids would also be the ones that loved it enough to go to the bootcamp. By this time, they'd actually be pretty good. They'd be working independently. Their mentors would review their work and help them get better, but they'd actually be doing stuff. This is how the business model would work. These kids would be worth sooooo much more than $10 / hour. They'd be doing client work that would get reviewed by the staff developers.

This concept isn't that crazy. It's not unlike how doctors and lawyers are trained. The only thing that's different is the age group. I think this has the potential to shake up two industries at once: software and education.

Right now this is just a thought exercise for me. I'd like to do it someday, but I'm in no rush. I'd love to hear your thoughts.

Ben:

Hi Zeke,

Sorry it's taken me a while to get back to you, there's a lot going on at work at the moment.

Before I get into a full discussion of your idea, I have a couple of clarifying questions:

With regards to problem one, what is the penalty for not doing anything about this? Why does it matter that programmers are overvalued and most people use digital tools without much power? This is not meant to be entirely rhetorical

What is the current and projected demand for programmers? Where is this data coming from?

I ask both of these first because you frame your idea as a solution. I want to make sure that we agree that there does, in fact, exist a problem; and it takes the form that you have described above.

Advocacy

An Example of Advocacy

I've decided to include an email conversation I've had with my high school Biology teacher.

Zeke:

Can I ask your thoughts on an idea I've been incubating? It's not something I'm likely to do now, but it has to do with teaching kids to code and I think you might have some interesting reality checks for me.

Ben:

Sure, ask away. I am happy to play the role of reality checker.

Zeke:

Problem #1: Good web development is too expensive. Programmers are in high demand and you have to pay top dollar to get a great-looking/working site/app. Most cannot afford this, so they use simple tools without much power.

Problem #2: We as a nation (planet?) aren't training programmers fast enough. This is one of the causes for problem #1. Programming can be taught effectively to children. It just isn't. In fact, teaching programming to kids has been anecdotally associated with higher test scores in math and science. Sometimes kids get so good at coding that they are ready for professional work in high school.

Solution to both: Let's think about a company. This company would make most of its money by doing web development contracts. We would bid 30% lower than the market, ensuring more reliable employment. We might even be in the position to pick and choose our clients. This part of the company would produce significant revenue because we'd take on more projects than we could reasonably handle (hang on, I'll explain how this could ever work in a second).

The company would also interact with kids in middle school and high school. We'd do development tutoring in very small groups charging hourly. This aspect of the business wouldn't be a cash cow either, but it would pay for itself. Kids who found they liked coding could go to an intensive development bootcamp over the summer. After the bootcamp, they would be eligible for apprenticeships. Apprentices would be paid $10 / hour and work a lot. These kids would also be the ones that loved it enough to go to the bootcamp. By this time, they'd actually be pretty good. They'd be working independently. Their mentors would review their work and help them get better, but they'd actually be doing stuff. This is how the business model would work. These kids would be worth sooooo much more than $10 / hour. They'd be doing client work that would get reviewed by the staff developers.

This concept isn't that crazy. It's not unlike how doctors and lawyers are trained. The only thing that's different is the age group. I think this has the potential to shake up two industries at once: software and education.

Right now this is just a thought exercise for me. I'd like to do it someday, but I'm in no rush. I'd love to hear your thoughts.

Ben:

Hi Zeke,

Sorry it's taken me a while to get back to you, there's a lot going on at work at the moment.

Before I get into a full discussion of your idea, I have a couple of clarifying questions:

With regards to problem one, what is the penalty for not doing anything about this? Why does it matter that programmers are overvalued and most people use digital tools without much power? This is not meant to be entirely rhetorical
What is the current and projected demand for programmers? Where is this data coming from?

I ask both of these first because you frame your idea as a solution. I want to make sure that we agree that there does, in fact, exist a problem; and it takes the form that you have described above.

I could give you my thoughts now, but that might be a little premature. Why don't you get back to me about those questions above and we can continue this dialogue

Zeke:

Hi Ben,

Don’t worry about how long it took you to get back to me. I understand how busy life can be.

I’m not sure you see programming the way I do. Let me try to show you how I think about it.

Language is an intrinsic property of humans; not a technology. We make languages like spiders make webs. Deaf twins in the absence of ASL education made their own sign language as their first language. They were on track (on track!) with their language development in terms of timing. Language is really a form of telepathy. I can put an idea in your head at my own will.

Writing on the other hand is different. It’s a technology. We developed it 5 or 6 thousand years ago as a way to make thoughts timeless, to transfer to others who weren’t there. It also had the added benefit of extending memory. People could write things down like how many chickens they had…

Writing is different from language because it extends humans’ natural abilities. If you put two kids together and don’t teach them, they will likely never be literate. It’s also much harder to learn as an adult. People have decided that this technology is so important that it was included in the UN Declaration of Human Rights.

So:

Language: innate, way of transferring ideas

Writing: technology, way of transferring ideas to different times, and remembering more

What are computer languages? Even if we didn’t have computers, computer languages would be useful on paper. They are very precise ways of writing down processes. They are specific and unforgiving. But, they completely describe a way of doing something that English doesn’t (intrinsically). The awesome thing about computers is that they can take this really specific language and DO IT FOR YOU! Every year we do more with computers. We communicate with them. We bank with them. We remember with them. We meet people. We even use them to think in ways that we aren’t inclined to, say, analyzing millions of records in a database…

Why are programmers in such high demand? There’s a computer in every office in much of the world. There are computers in most people’s pockets. We have this fantastic foundation for so much. But, very, very few know how to build.

I’m not saying we should replace learning to read with learning to code; that’s crazy. I’m just saying that it’s not even offered in most high schools. They have computer classes, just not coding classes. This is crazy. It’s also pretty crazy to insist that kids memorize the Pythagorean theorem, the Krebs cycle, and have no understanding of the backbone of modern math and science: computers. For more on this: Code.org Video

Also check out this infographic that cites the Bureau of Labor Statistics, the College Board, and the Association for Computing Machinery: Code.org Stats

Ben:

Hi Zeke,

Thank you for explaining your thoughts in more detail. I think you make a compelling case for the importance of programming skills and the potential for integrating them into education more broadly. Here are my thoughts:

I agree that there is a significant value in teaching programming to kids. It equips them with a new form of literacy that is increasingly relevant in our digital world. The analogy to language and writing is particularly striking.

Addressing the demand for programmers is indeed a critical issue. The data you’ve provided shows that the demand is growing and that we’re not keeping up with it. This mismatch creates opportunities for initiatives like the one you proposed.

Your idea of a company that combines web development services with educational programs is innovative. It leverages the high demand for programming skills to create a sustainable business model while also addressing an educational gap.

One potential challenge is ensuring the quality of education provided to the students. You’d need a strong curriculum and experienced mentors to guide them effectively.

Another consideration is the scalability of the model. How would you ensure that the educational component can be scaled up to reach more students without compromising on quality?

Overall, I think your idea has a lot of potential. It addresses real problems in both the software industry and education, and it could have a significant impact if executed well. I would be happy to continue discussing this and help you refine the concept further.

Conclusion

I predict that when we look back we will see the early 21st century as an unusual time. Computers have become commonplace. Most aspects of our lives have been transformed by their presence. Yet, most people don’t have the ability to harness their true power.

Often organizations conclude that they can’t build their own computer systems. They hear about tech companies hiring hundreds of programmers and paying them well. But organizations can’t ignore computers either if they want to interact with the world. Organizations feel forced to use “canned goods”—computer tools that are supposed to “work” out of the box. I put “work” in quotes because they don’t really. They serve the lowest common denominator and take away many of the intrinsic benefits of computing. They don’t share data well. They don’t promote structured thinking (or data). One example of this is covered in detail in my opinion piece. I don’t feel the need to add more detail here.

My Div III shows that there’s another way. I worked with (mostly) non-techies to build “empowerment software.” I showed that it’s possible to build web services that make a huge difference quickly and (relatively) cheaply.

Much of the work done at tech companies is handling problems of scale. If Facebook was built to serve 1000 people, it would require a fraction of the development power, even with the same feature set. The problems these organizations face typically don’t need to be brought to millions. They can hire just one developer. If they did, a whole slew of new possibilities would be open to them—improving efficiency and quality of work.

In fact, empowerment software is getting easier and easier. More and more services are open to small developers. Twilio is a great example of this. It lets someone with basic programming ability write web apps that can make and receive phone calls and text messages.

But, more work is needed. There aren’t enough programmers. This makes even hiring one too expensive for many. If you ask me, something this important to society should be taught in every high school. At least basic competency should be a requirement. Changes like this would empower countless people and organizations to live better.

Await... async

2015-11-15T00:00:00Z

Sequelize ERD

2016-01-18T00:00:00Z

ERD diagrams can be awesome. They visualize the relational graph of your application's data model. If your app is small enough, they can be a great piece of documentation. But, it can be hard to keep them up to date. That's why I made SequelizeERD. If you use Sequelize, plug it into SequelizeERD and you'll automatically generate diagrams in whatever format you'd like.

Installation

You don't need Graphviz or any non-javascript software to use this. Just

$ npm install sequelize-erd --save-dev

API

Exported from sequelize-erd is a function which takes your models. It can either take the sequelize instance or a path to a file to require. The function returns an svg of the models.

const { writeFileSync } = require("fs");
const Sequelize = require("sequelize");
const sequelizeErd = require("sequelize-erd");

(async function () {
  const db = new Sequelize(/* Your Sequelize config object */);
  // Import DB models here

  const svg = await sequelizeErd({ source: db }); // sequelizeErd() returns a Promise
  writeFileSync("./erd.svg", svg);

  // Writes erd.svg to local path with SVG file from your Sequelize models
})();

You can also customize the output format, engine, arrow shapes, arrow size, line colors, and line width as well as include/exclude specific models.

const svg = await sequelizeErd({
  source: db,
  format: "json", // See available options below
  engine: "circo", // See available options below
  arrowShapes: {
    // Any of the below 4 options formatted ['startShape', 'endShape']. If excluded, the default is used.
    BelongsToMany: ["crow", "crow"], // Default: ['none', 'crow']
    BelongsTo: ["inv", "crow"], // Default: ['crow', 'none']
    HasMany: ["crow", "inv"], // Default: ['none', 'crow']
    HasOne: ["dot", "dot"], // Default: ['none', 'none']
  },
  arrowSize: 1.2, // Default: 0.6
  lineWidth: 1, // Default: 0.75
  color: "green3", // Default: 'black'
  include: ["artist", "song", "album", "artistSong"],
}); // sequelizeErd() returns a Promise
writeFileSync("./erd.svg", svg);

From bash

Options

source relative path from project root to js file containing sequelize object with models loaded
destination Where you want your ERD SVG
include Only include the following models
exclude Exclude the following models
format File format. Options are "svg", "dot", "xdot", "plain", "plain-ext", "ps", "ps2", "json", "json0"
engine Layout engine to use, options are "circo", "dot", "fdp", "neato", "osage", "twopi". Default to "circo"

We expose a binary for you to use as a npm script. If you want an erd diagram generated in your project folder every time you commit, add this to your package json.

The source path specifies a js file that must export your Sequelize DB instance. It also needs to load all your models.

{
  "scripts": {
    "erd": "sequelize-erd --source ./path/to/your/sequelize/instance --destination ./erd.svg"
  }
}

We have coding in schools backwards

2017-05-21T00:00:00Z

To hell with the job market: Kids should code because it helps them think.

The premise was simple. In 2013, If you can code, you'll be able to do things no one else can - be a wizard, or a rockstar. Different strokes for different folks - the point is Zuck can't hire enough engineers, so he really needs you to get started quickly.

I have to admit, I was into it. I learned to code as a kid. I started with webpages. With the help of mentorship, I was writing software by the time I started high school. Coding shaped how I thought about everything. I made a database driven website for vocabulary words in English. I made visual history timelines. I modeled physics homework in PHP (side-note: not the best tool for the job).

The idea that coding was coming to the masses and changing education was (and is!) tantalizing.

In the years since, my perspective has changed. The video has it wrong. We should indeed teach kids to code - but it's not about jobs - It's about thought.

Let's play a game. How many trade relationships are there between countries in the world? Let's say there are around 200 countries, and most countries have relationships with most other countries… how many could it be? 1000? 5000? Take a moment to guess - I'll wait.

The actual number is close to 20,000 or 40,000, depending on if you count a bilateral relationship once or twice.

A modern high school student writing a paper about trade would have no practical way to get a sense for the system of global trade. Sure, she could pull up facts about what the United States and China trade with each other. She could quote the White House and congressional Democrats on their opposing views of tariffs. But, she wouldn't be able to answer any of her own questions because of the information bottleneck of tens of thousands of data points.

You may be thinking to yourself that this kind of sophisticated data analysis is beyond the grasp of a high school student - that an economist or statistician should be asking these questions. You may even be correct in 2019, but it doesn't have to be this way. These types of questions are within the intellectual grasp of young adults, and the tools exist to let them interact with them.

It was for this reason that Steve Jobs called computation the Bicycle of the Mind. Computers makes it possible for everyone to play with information, answer their own questions, and discover things about the world.

Let's play another game. Let's compare Drivers' Ed. to actually driving a car for the first time.

In Drivers' Ed., students learn about the speed limit and the dangers of drinking. They learn about state laws and the consequences of breaking them. In a good class, they might learn a little bit about how cars work. Students do not learn how to drive.

Learning how to drive a car is exploratory. You can literally take any road! It's a little scary. You need support and guidance, but at the end of the day, it's the student's job to try something, experience what happens, and learn from it.

Today's conventional high schools are more like Drivers' Ed. than driving. Students are asked to take in and regurgitate a tremendous amount of information - everything from the citric acid cycle to the Pythagorean theorem - in the hopes that this knowledge will expand their intellectual horizons. It even works, to an extent. It just doesn't come anywhere near its potential.

We should be empowering students with the tools to become the next Pythagorus, or the next Albert Szent-Györgyi.

Teaching the process of intellectual exploration is a lot harder than teaching facts. When you teach facts, you can expect a certain hegemony of outcomes. 35 students, 5 outcomes: A, B, C, D, F. Students knew the enzymes required for photosynthesis or they didn't. Student's could "do" the stoichiometry, or they couldn't.

But did their minds expand?

In an exploratory environment every student is expected to have their own unique outcome. The outcomes are hard to compare to one-another, but that's not really the point. Students probably don't have textbooks, workbooks, tests or homework. They have access to the raw tools, and a guide that can help them start to answer their own questions. Topics can be introduced. Teachers can help students get back on track by guiding them away from a line of inquiry, or critiquing how they used a certain tool.

The value of inquiry-based learning is clear when you look at the macro outcomes. Citizens who have a fuzzy memory of how photosynthesis works don't change the world. People who discover the next Photosynthesis change the world.

Of course knowledge is important. Of course facts are important. Hard skills are important, too. But, all knowledge comes from inquiry. All facts start as questions. Schools should be a place where students learn how to inquire, play, and create.

There's no better tool to inquire, play, and create than the computer.

What would STEM education look like if the ability to program a computer was a given?

Offer scorecard

2018-11-15T00:00:00Z

Link to model: model discussed in talk

Pandemic writing practice

2020-08-29T00:00:00Z

Living through a pandemic is rough. I've been thinking about how I spend a lot of time consuming content. After work, I sit on the couch and look at netflix, tiktok, twitter... I refresh the new york times and washington post. I look at hackernews. I'm chasing that dopamine hit.

The brain is a plastic organ, changing over time in response to stimulation. I really enjoy when I'm thinking actively. I enjoy producing. But, just like with my body and exercise, when I don't do it enough... it gets harder.

Don't get me wrong, I spend a lot of time thinking. My job is cognitively demanding. But, I feel like I'm not being as stimulated as I could be across the board.

To break this down, I'd like to start by exploring input to output ratios of various activities. In this exercise, I think it's important to try not to judge myself or others. It's hard. I probably will. When I write the word "TV," I get into a very judge-y, finger-wag-y headspace. I'm not sure that's productive.

Input: like stdin. It's things I take in from the world

Output: like stdout. Talking, writing, drawing, even moving maybe?

I'd like to look at these few example activities with another axis: "CPU load." How much do they make me think?

I want to talk about how I spend a typical work day since COVID.

wake up
feed the cats
turn on NPR
have 2 espresso
read all news sites
Try to get revved up for the day... try to see it as exciting, try to understand what's going to happen.
This is easiest when I have a feature i'm excited about or a deadline. It's hard when there's a lot of meetings or i feel like i'm trudging through something exhausting
Start working
Having meetings
Do more work
Look at twitter
take a walk
At some point lunch
More work
twitter
More meetings
At some point work stops.
Start thinking about dinner
Watch some tv
cook sometimes
watch some tv
twitter
20 min on the exercise bike while watching tv
shower
tiktok

Aside about noticing gaps here

An interesting observation about the chart above... By my own assessment, I don't really have any high output mindless activities... I'm not sure what they would or could be....

What's getting me motivated to make changes in my life is all the "low CPU" input tasks my brain is doing after work. Aside from talking to claire over dinner, I feel like I'm "vegging out." It has a very brain death feeling^[1]. Tweet by tweet. Each requiring no critical thought.

I also think it'd be nice to find things to do when taking breaks... I feel like i'm doing a lot of ADD-esq refreshing of COVID numbers.

It's not like I think you should be outputting all the time... It's not like I think you should be in deep number crunching all the time. I think you need a healthy balance! I think it's good to have time to rest. But, I think my outlet for processing stress has been to consume massive amounts of thoughtless media. I'd like to change that.

So what am I going to do? Well, for one, I'd like to start a writing practice. I want to write every day. Not something for others to read. Just to get some output. I think it'll help me think, but I also think it'll be good for my brain. Just like exercise. I'd also like to read more... we'll see where that goes.

I'd like to have a dashboard. It can be spreadsheet for now. It just tracks if I'm doing OK with exercise and writing... or maybe sleep, exercise, healthy balance of IO and CPU load.

I think i need to find some passive input that is emotionally salient too. Like a good fiction book to dig into. Something to do before bed that has me thinking, retaining state

Another thing I'd like to discuss is that there are times where there is no input and no output but there's still CPU activity or memory access. Think about a long walk where you're thinking through something. You're not producing. You're not consuming. Or maybe you are and the read and write streams are just pointed at your brain.... I think that time is important to. I don't make enough time for it. All of this activity ultimately comes from that kind of time

people who watch more hours of TV have faster cognitive decline ↩︎

Using Vega Lite to explore chemistry

2021-02-28T00:00:00Z

Check out this post on Observable

Playing with Hilbert

2021-03-13T00:00:00Z

Credit: Playing with Hilbert by Zeke Nierenberg

Cost of computing

2021-10-29T00:00:00Z

Before we talk about the cost of computing, let's talk about the cost of computers.

I did a comical estimation with Claude on the cost (in todays dollars) for the CPU time that would have been required to load a web page in 1960 if the web had existed. It was $3,800. Today the same web page can be rendered for $0.00000006.

1957: The first municipal computer (Norwich, UK). Just one of 21 cabinets making up the computer.

2015: The Raspberry Pi Zero

Credit to Steven Pemberton for this comparison

But, it's not just computers. It would have cost more than $10k (in today's dollars) to light your home for a day several hundred years ago.

We accept that prices change throughout history, but the sheer magnitude of the decline in technology prices is staggering. People in general are not great at contemplating non-linear changes^[1].

The Experience curve effect is a pattern that shows up repeatedly in technology. As people learn, they get more efficient. They drive down errors by improving processes. Supply chains are optimized. Then, second order effects start to happen. As prices drop, markets grow. Purchasers who were recently priced out are start buying. That allows economies of scale come into play.

Efficiencies compound. If you take a nap for a decade or two, numbers can get very very 🤏. Entrenched market dynamics that were once a given become a historical footnote. For example, it's not uncommon to hear people marvel at the fact that computers used to take up entire floors of high rises. Go to cocktail parties and start listening for the word "before." The internet before the web. Personal computers before GUIs. Solar. Light. DNA Sequencing. Aviation. Automobiles. Batteries. The list goes on and on.

Inflection points

Experience curves follow a nice predictable path until they don't. At some point the curve breaks. And when it does, it happens in one of two ways.

Plateaus

Plateaus occur when some kind of local minima is reached. For example, people have been saying we're almost at the end of Moore's law for my entire life. Even Gordon Moore predicted that Moore's law would end at some point. At some point the laws of physics impose their limits. You can't really make a transistor smaller than an atom.

"Earthquakes"

Sometimes the set of forced governing the experience curve suddenly change. This could be a technological innovation, government intervention, or even ecological or economic or cultural change. Whatever the cause, when a new dynamic takes effect, prices can change even more suddenly than an Experience Curve.

In the case of whole genome sequencing, an inflection point occurred when Massive Parallel Sequencing replaced Sanger Sequencing. It took this technology years to get to market, but once it did, the Experience Curve broke and a new one formed.

It's hard to appreciate what this can mean for society. Every time we pay for something we're deciding not to pay for something else of the same price.

I asked Perplexity to help me come up with something else society could have done with the money it would cost to sequence one person's genome over the last 24 years.

Year	DNA Sequencing Price/ person	Something else society could have done with the money
2001	$100,000,000	Expansion of a (non-US) public transit system (e.g., Metro de Los Teques in Venezuela)^[2]
2006	$10,000,000	Renovation of a public library branch^[3]
2008	$1,000,000	Installation of a traffic management system^[4]
2011	$10,000	Purchase of a police patrol vehicle^[5]
2015	$4,000	Installation of playground equipment^[6]
2020	$1,000	Purchase of a commercial-grade AED^[7]

Different things are possible when something costs $10^8 than when they cost $10^3.

What about the cost of writing programs?

When I say "cost of computing," what I mean is the units of effort are required to express one unit of computational thought to a computer. We can sort of think of that as a price... And in many ways it is. Sometimes it's a cost in literal ways: it's an input to the cost structure of any organization employing software engineers. Sometimes it's a cost in less literal ways. When the cost of expressing computational thought changes, the ambitions of my side projects change because it's suddenly possible to do more.

For the last couple decades, while I've been a programmer, and until quite recently, the cost of computing has been in a plateau. Programming is different than it was in 2003. Programming is better than it was in 2003! But, (again until recently!), programming is not cheeper than it was in 2003, in my opinion.

But, historically, we know that at one point writing programs used to be more expensive than it is today. There was a time, for example, when programs were written on punch cards! For ages programs were expressed in assembly. It's unclear to me when the price started to stagnate.

The predominant UI for programming has been a bunch of text files in folders that get compiled down to something deeper. It's been a long time since a really radical programming idea has gained wide-spread adoption and created a new dynamic. I'm not talking about some neat programming language. I'm talking about a different way of thinking about what a program even is. One I can think of is the Spreadsheet.

Have you ever seen a spreadsheet expert at work? It's magical. Spreadsheets–when leveraged to their full potential–let the programmer construct elaborate computational models. They're live and reactive, not text files. Data visualization is built in. The learning curve is smooth. I think that a smart novice could start with recording tabular data and get them self to Monty Carlo simulations in about a month.

And they took over every industry. Every organization I've interacted with is dependant on an intersticial network of inconsistently maintained spreadsheets. I'd like to know if there's a single organization today that is not run this way.

Spreadsheets are so good that even when Microsoft made their UI hideous for a couple decades, they still took over the world.

I'm not aware of any programming environment since spreadsheets that have as fundamentally altered the cost of composing programs, and none that have enjoyed widespread adoption.

It's worth taking a beat to appreciate what I just said. How is it possible that the last major inflection point in computing's cost curve happened in 1979 on an Apple II?

Of course you can't run web servers in spreadsheets. You can't build Sim City in spreadsheets. Spreadsheets aren't even turing complete without some scripting language. So while spreadsheets altered the cost of computing radically, on their own, they were limited to a certain category of program.

This visualization was created using v0.dev

What are the factors that contributed to the success of spreadsheets? Here's my list.

Very low barrier to entry. They produced value for all users, regardless of technical background.
Everything is a cell. One abstraction to encapsulate both data and code. When you reference a cell, you don't care if it's a data cell or a code cell. It's a cell.
Runtime === buildtime === prod. This thing is live! You didn't need to understand or debug some pipeline to get the thing to do something. Just open a file and you have the real thing.
They were playful! Just edit a cell and everything reacts.

Let's contrast this with Smalltalk

Smalltalk ~~was~~ is awesome. Admittedly, as a 90s baby, I never really got to play with it in it's heyday. While Smalltalk never really broke out into mainstream society (ask the normies in your life about it, they'll blink at you), it'd be crazy to call it a failure. It's influence on programming writ large cannot be understated.

Smalltalk was the first truly object-oriented programming environment.
And the first instance (correct me if I'm wrong) of a programming environment running the program and giving the user feedback (like spreadsheets!).
It was alive–no build step.
Everything is an object. One abstraction to encapsulate everything from data to code.

The UI influenced modern operating systems and IDEs. In 1979, Steve Jobs famously ripped off the Smalltalk UI for Lisa when he was shown it running on the Xerox Alto computer.

While spreadsheets and Smalltalk were both reactive programming environments, Smalltalk just was just too expensive to be viable as an end-user programming environment. One of the project's stated goals was to be approachable to children. It wasn't. If it were, more children would have used it.

In order to use Smalltalk, you had to write code. Code takes effort to read and write. It's fickle. Humans, for the most part, struggle to reason about it without experience. Unlike a spreadsheet, you can't just open up Smalltalk and fuck around and find out–at least not without a lot of effort.

Both Smalltalk and Spreadsheets are governed by a single abstraction: For spreadsheets, the cell, and Smalltalk the object. One reason for the success of the spreadsheet is that it's single abstraction was tangible to a novice user. The building block of the whole damn thing had a UI.

Hypothesis: we can predict the success of a computing environment by judging the quality of the abstractions and the quality of their UX.

Before LLMs, which I'll get to, the last time I got excited about a piece of technology maybe breaking the Computing Experience Curve was in 2013 or 2014 when I tried Airtable. Airtable took a powerful software pattern–the relational database–and put a next level UI on top. High quality abstractions. High quality UX.

Airtable showed so much promise. Everyone already used Google Sheets as a makeshift database - sharing lists where each row followed a consistent structure of columns. The familiarity of spreadsheets made it easy for anyone to contribute data and leverage formulas. But spreadsheets weren't built to be databases, even if you could awkwardly force them into that role. But, it was the best tool for the job because there was no one^[8] in the market making an end-user facing database that made any sense.

The abstractions of a relational database (tables, rows, types, relations, constraints, etc) are more complex than the abstractions of spreadsheets. But, I thought that Airtable's UI was so good that maybe it could effectively teach novice users how it worked through experimentation. It seemed a hell of a lot easier than anything else out there.

erDiagram
    ARTIST {
        int artist_id PK
        string name
        string nationality
        string birth_year
        string death_year
        text bio
    }

    ARTWORK {
        int artwork_id PK
        int artist_id FK
        string title
        string year_created
        string medium
        string dimensions
        string image_url
        boolean on_display
    }

    COLLECTION {
        int collection_id PK
        string name
        string description
        date creation_date
    }

    GENRE {
        int genre_id PK
        string name
        string description
    }

    ARTIST_GENRE {
        int artist_id FK
        int genre_id FK
    }

    ARTWORK_COLLECTION {
        int artwork_id FK
        int collection_id FK
        date added_date
    }

    ARTIST ||--o{ ARTWORK : creates
    ARTIST }|--o{ ARTIST_GENRE : belongs_to
    GENRE }|--o{ ARTIST_GENRE : has
    ARTWORK }o--o{ ARTWORK_COLLECTION : included_in
    COLLECTION }|--o{ ARTWORK_COLLECTION : contains

Airtable took a leap of faith and trusted that the bigger market would take the time to learn the way Airtable worked. And in return, Airtable created a set of well designed, well thought out, inviting power tools. They had a generous free plan so people like me could try it (and then advocate to use it at work).

And people like me loved it.

I used Airtable to run my wedding's invite and RSVP system. I used Airtable to build #walkthevote in 2020. For our wedding, we had tables like households, invites, rsvps (which had dietary preferences), gifts (for keeping track of thank yous). We had almost everyone's cell phone number, but we didn't have a lot of physical addresses. I wrote a web service to text everyone and have a dialog with them to gather their address and insert it into the address table.

Unfortunately for Airtable, Inc, while their UI is awesome, it's not awesome enough to make this concept understandable to a novice user before they give up. Some users embraced the tool and put in the work to learn it. But that market just wasn't nearly as big as Airtable, Inc had hoped. To be clear, Airtable is a successful company, and if you ask them, they'll tell you that it's awesome that they've sold into so many fortune 500 companies. But they are not on track to become a verb, change the cost curve of computing. They're "just" normal innovation.

One afternoon in August, I remember nearly doing a spit take when trying out a new thing from Supabase.

So we're finally getting to LLMs.

☝️ This thing (called database.build but previously called postgres.new) is so fucking cool.

It's exactly what Airtable was missing in their first-time UX.

On some level it's "just" another browser-based agentic code generators not unlike Claude Artifacts or OpenAI Canvases or V0. In all of these cases, a runnable artifact is produced and evaluated as you chat back and forth with the LLM. But with database.build, the code takes a back seat in the UI. Instead of reviewing code, you review a pretty sick diagram made with react-flow.

Database.build runs a real postgres database inside the browser with pglight. If a DDL statement produced by the LLM doesn't work in the real database, the LLM gets that feedback and tries again. It's typically able to self correct. If you had asked me in 2016 when this kind of technology would come to the market, I would have said "never."

Here's basically how database.build works:

flowchart LR
    User((User)) -->|"Natural Language Request"| LLM

    LLM --> DDL[SQL DDL]
    DDL -->|"Generate visualization"| Diagram
    Diagram -->|"Reviews"| User
    subgraph WASM[WebAssembly Runtime]
        DB[(Postgres)]
    end
    DDL -->|"Execute"| DB
    DDL --> LLM


    linkStyle 0,1,4 stroke:#e74c3c,stroke-width:2px;
    linkStyle 2,3,5 stroke:#2ecc71,stroke-width:2px;

There are two general categories of cost when developing software: reading code and writing code. Here You write in english and you read a diagram. When things get off the rails, you can open up the DDL and fix it.

In the future, the user might not need to know about the DDL at all.

I wouldn't be surprised if Supabase ends up being the company to build the first database that the normies actually start using, although that doesn't seem to be their mission. Airtable tried to make creating a database easier by building fancy point-and-click UI. Supabase is making it easier by using an LLM (which, to be fair, wasn't available to Airtable when they started). Their core units of abstraction are the same, but their presentation and interaction layers are totally different.

Hypothesis: One big reason this thing works as well as it does is because of it's constraints. There are only so many things that are possible in a relational database! Contrast this with V0/etc. The universe of possibilities is far far wider, and that makes it harder to get just right. It also limits what's possible with database.build. You can't build a 3d visualization in a database, after all.

None of the projects I've talked about thus far are "real" programming environments. They're all toys. You can use them to build impressive stuff, but you don't maintain real Big Kid Software (todo: trademark "Big Kid Software"). in them. Of course.

That brings me to another pattern that's very hot right now: IDE + AI. Examples include Cursor, Zed, Copilot and more.

Cursor is an interesting case that I should discuss briefly. I use it. It's good. Quite good, even. It's kind of incredible that they've continued to lap microsoft over and over again using their very own open sourced IDE as a foundation. Copilot, Microsoft's version, started first but somehow seems to be constantly playing catch up. It's not bad, it's just not as good as Cursor.

All these IDE + AI apps are examples of AI veneer. Take a product that's already good, slap some AI on top. There's nothing wrong with veneer. Veneer is nice. You can improve a product with AI veneer. You can even improve it a lot! You can probably use AI veneer to reduce the price of programming by some meaningful amount. But. I promise you it is a trap headed to a plateau. These apps are going to make it possible for product managers to make really cool prototypes. They're not going to change the nature of software.

It'd be a tragedy if in the 2060s we were still coding in text files on little screens with keyboards and LLMs.

We are having a mental cache invalidation event and we need to revisit our priors. Every programming environment idea we've ever thought of that was "awesome but impossible" or "science fiction" deserves another look. Having fuzzy (even flawed), intelligent building blocks in our computers opens up the possibility for a truly different programmable medium.

Of course there are people working on this. Lots of them. I know some. The folks at TLDraw are doing some really interesting stuff. In theory they're a collaborative whiteboard company. And their flagship product is quite good. But they seem quite preoccupied with thinking about computing and computational media. In 2023 they made MakeReal, a tool that let users draw a UI mockup and get a working prototype–sort of like V0 but without text prompting. Then recently they announced TLDraw Computer

I'm not saying we're all going to be writing programs in TLDraw Computer in several years. I don't think TLDraw is saying that. I don't think you could write TLDraw in TLDraw Computer. But. They are doing the work. They're rethinking the foundational assumptions. And I'm really eager to see what's next.

I think we very well might be in an inflection point in the Efficiency Curve. We probably are. The question is, how low will we go. If we stop at AI assisted coding, and lose sight of the dream of AI assisted programming, we're at risk of getting caught in a disappointment local minima. Instead, I'd like to see a programming environment that

a few high quality, powerful core abstractions
very very good deeply intelligent UX on top of those core abstractions
Is not a toy - you should be able to build modular units and hide complexity
Is visual: end user sees and manipulates some visual representation of the program, not just code
Exposes contracts and types as a first-order - we haven't really touched on this but it feels important
Semantically searchable: "find where we make the api call to X" should be as easy as grepping
Local first (a guy can wish right?)
Fast (goes w/ the above)
Fun

this post is still under construction

scrapbox

	Spreadsheets	Smalltalk	Conventional Coding	Airtable	Database.build	TL Computer
Core Abstraction	Cell	Object	...	Table/Relation	Table/Relation	Node
Core Abstraction UI Quality
Instant Feedback
Entry Barrier
Complexity Ceiling
Scale Ceiling
η Ratio (Effort/Thought)
Domain Limitations

stuff to weave in

https://river.maxbittker.com/
Making good software requires taste, craft, prioritization. Making software that changes the world requires new abstractions.
Obsidian
Dynamicland

airtable evil?

I can't talk this much about Airtable in a blog post without telling you that... they might be evil.

Recently, when I needed to access our wedding Base on Airtable, I was warned that since the last time I had used it, a new artificial row limit had been imposed of 1000 rows. That's insane. And stupid. If you're going to build a personal computing tool, and you're going to offer a free version to consumers, then you need to be able to plan a wedding in. I get charging companies for SAML integration, auditing and collaboration tools, support, etc. Makes perfect sense. Don't offer consumers a free plan if you can't plan a wedding in it. It forces last minute Saturday sqlite migration scripts. And that's annoying.

And. If you're going to impose a stupid, pointless row limit, you gotta let our existing bases keep working.

OK. That might be enough ripping into Airtable. They're still cool. But I'm not using them anymore. If you're looking for recommendations, try Baserow or Nocodb. Or, just use sqlite.

Exponential Growth Bias is the formal term for the phenomenon that humans seem to understand and predict linear changes far easier than exponential ones. It's why people had a hard time accepting COVID at first. How could ~30 turn into millions so quickly?? ↩︎
In 2001, $100 million was allocated to the Metro de Los Teques project in Venezuela to expand public transit and connect Los Teques with Caracas. Source: CAF Development Bank Report. ↩︎
The Seattle Public Library renovated its Yesler branch in 2006 for $6.8 million as part of its "Libraries for All" bond issue. Source: Seattle Public Library Renovation. ↩︎
Advanced traffic management systems cost between $500,000 and $4 million. For example, the I-90 Active Traffic Management system in Washington State cost around $12.9 million in 2010. Source: Washington State DOT Report. ↩︎
Police patrol vehicles typically cost around $10,000 to $12,000 per unit in 2011. Source: Rio Rancho Police Capital Improvement Plan. ↩︎
Playground equipment costs vary but individual pieces can range from $3,000 to $5,000. Source: Playground Equipment Cost Estimator. ↩︎
Commercial-grade Automated External Defibrillators (AEDs) typically cost around $1,000 to $2,500 today. Source: AED Superstore Pricing Guide. ↩︎
There were some other end-user-facing databases. One I remember messing around with as a kid was Microsoft Access. I have a vague memory from the 90s of my mom working on a DOS database with my dad as they set up the infrastructure for her Greeting Card business. None of these technologies enjoyed widespread adoption, though. And it felt like Airtable really could. It had mobile support. It had tons of cool web tools. ↩︎

How to help a student get unstuck

2021-10-30T00:00:00Z

When I was eight years old, I had a profound realization: once you know how to read, it’s impossible not to. If you see the letters c-h-a-i-r in sequence, your mind conjures sit-able 4-legged furniture. You don’t choose to read or not read, you just do it. And it’s impossible to understand what it’s like not to be able to read.

The same principle applies when learning how to code. Once you understand how it works, it’s extremely challenging to imagine not understanding it. This can be problematic when teaching because it creates a gap between teacher and student. Even the most empathetic teachers can’t quite imagine themselves in a student’s shoes^[1].

Getting a student from naivety to mastery takes a lot of work from the student and the teacher. But the work is asymmetrical. As a teacher, your role is to accompany, you can’t walk the student’s path for them. All you can do is support them through the journey as best you can.

This article is intended to share some techniques I’ve developed for teaching debugging in a one-on-one setting. ~~If you find this useful, let me know, and I’ll write another one about helping students who are pair programming.~~ I wrote a follow up article discussing dynamics in pair programming.

Forget about the program, focus on the programmer

Most programmers enjoy programming, and love solving a good problem. When you’re helping a student write a program, you have to fight the urge to fix the problem. Forget the program is even there. You may just want hop through a few files and figure out what’s going on. Stop. Fight that urge.

While it can be marginally helpful for a student to see you at work, it’s not nearly as useful as it would be to help them solve the problem. You’re not there to get the code to work—that would be consulting. You’re there to get the programmer to work.

As programmers we construct an abstract mental model of the program we’re writing. Bugs are discrepancies between our mental model of the program and the program itself. The program is doing something we don’t expect. The program isn’t wrong. The program can’t be wrong. It’s doing what it’s been told to do. What’s incorrect is our mental model. Once we understand the current state of the program, we can adjust it.

The art of debugging is systematically and creatively synchronizing the mental model with the actual program.

As you teach, your focus should be on the student’s ability to keep the two in sync, not your own.

Don’t let them overheat

Debugging can be frustrating. Every programmer knows the difference between productive friction and wheel-spinning. Your student can’t always tell the two apart. It’s your job to keep an eye on their frustration level, and adjust your support strategy accordingly.

Consider both the frustration level of the student and the instructional value the moment offers.

If they’re getting stuck on some tiny little problem, and the answer’s starring them right in the face, just fix it for them. It’s not worth pulling out the socratic method to fix a missing comma.

If they’re profoundly frustrated, even if the moment offers tremendous instructional potential, hit pause. Take care of the human being in front of you. The programming concepts can come later.

Depending on the level of trust you’ve built up with the student, you may be able to teach through more frustration than you would with a brand new student.

So, You’ve identified a strategy. But how do you teach, how do you support and how do you debug?

How to teach

Try to ask questions more often than you give information. It’s totally fine if you tell students things here and there, but try to lead them to answers whenever possible. I’ve found that sometimes I can deliberately slow down my own thought processes, and shield myself from what I think is the right answer. By doing this, I’m able to pose questions that I don’t know the answer to (“I wonder what the state of the user record is right now?”).

If you’re teaching in-person, avoid touching a student’s keyboard. When I was new to teaching, I would sit on my hands so I wouldn’t be tempted to take over. If you need to show an example of something, use your own computer.

Offer lots of encouragement. Every idea is a good idea. If they’re at risk of rabbit-holing, gently coax them out of it with a question, observation, or suggestion, for example, “that’s a great thought… I don’t think we’re going to find the answer there… tell me, do you think this code ever runs? I’m starting to wonder if the problem is upstream.”

When you get a sense of closure on a topic, prompt the student to summarize what they’ve learned. Finding the right time to summarize isn’t totally cut and dry, but I’ll offer two metaphors. When the “mental callstack” is empty, summarize. When get that feeling like “I should make a git commit,” summarize. Students sometimes get overwhelmed by the details, and if they never take a step back and appreciate the forest, all they’ll remember are trees.

How to debug

So, you’ve decided that the best course of action is to take over and solve a student’s problem for them. The way you do it matters. You want to make sure you’re honoring the weight of the problem. They’re not stupid for not getting it. Show them that even professional programers get stuck, and that there’s a methodology for getting unstuck.

It’s important that they can follow along. You might be making certain observations--voice them, and then check to see that they’re with you (“oh, it can’t be a problem in the checkout service because we haven’t even passed validation, yet… do you agree?”). Make lots of diagrams.

Practice good hygiene. Even if you might throw a bunch of random logs all over the place when you’re coding alone, try to demonstrate best practices. You don’t want to model your own bad habits.

When it’s fixed, go over everything again. Talk about what the problem was, how we figured it out, and how we fixed it. Once you’ve done that, check and make sure they’ve got it.

How to support

Some moments are “teachable” and some aren’t. Learning to code is hard, and you will find students who are so frustrated or anxious that they can’t learn. This isn’t the time to pivot to your favorite debugging technique. Instead, be there for them. Hear their concerns and anxieties, and let them know that they’ll get through it.

If you’re in a setting around other students, take them aside. It can be hard to have a vulnerable conversation in public. Maybe take a walk or go into another room.

The most important thing you can do is listen. Ask them what’s bothering them. Make sure they know that you heard them by repeating it back in your own words. If you don’t have it right, keep working at it until you do.

Lots of early stage programmers don’t realize that it’s normal to struggle. They may not be aware that you had analogous struggles. While this isn’t about you—it’s about them—it can be helpful to tell them about the ways you struggled while you were learning, and the ways you struggle today.

It can be hard for students to grasp the progress they’re making when they’re in a rut. At times, I’ve pulled up prior work from the student and emphasized the growth they’re making. “I know how frustrating this is. I just want you to know that you’re growing, and I see that. See this? You wrote this two months ago. I bet you’d do it totally differently today. What about this? You did this last week, you couldn’t do that two months ago.”

Taking a break is almost always necessary before diving back into the IDE.

Putting it all together

You can mix and match these techniques. You can throw some out and add your own. It doesn’t have to be 100% one and 0% the others. You’re not a machine. Every teacher and every student is different. The art of teaching is finding a technique that works for the situation at hand.

Can’t tell if it’s working? Ask your students. “When we took a walk and talked through things, was that helpful?” “When I ask you all these questions as you’re coding, do you find that useful? I sensed some frustration.” Just like you can’t know the state of an application without a breakpoint or log statement, you can’t know the state of your students without talking to them. What they say may surprise you, and it’s important for fine tuning your craft.

@Ozzie_osman on hackernews told me that there's a formal term for this: The Curse of Knowledge ↩︎

Zeke's new brand rollout

2021-10-31T00:00:00Z

I'm pleased to announce today that I'm rolling out a new version of my personal brand, visual identity, and website. This is the culmination of a lot of work by my internal marketing function, and it's absolutely something to celebrate.

New headshot

Old headshot from 2019

Today's new headshot from the inspired Be Nice Comic

New logo and name

There was no logo or brand name

off-by-one

Brand new name and logo

Inspired by one of the most common programing errors, off-by-one evokes the sense that I'm just a little bit out there... that a couple screws might be a little loose--a reality that I'm embracing today.

My internal design function burnt the midnight oil considering many alternate logos before finally landing on one that captured my ethos.

Why I refreshed my brand identity

My prior identity was established in early 2019. Today, we live in a different world. In 2019, I was attempting to appear professional, considered, and deliberate. But, the feedback from the community was that the brand was a touch boring. We heard you. With today's refresh, you'll find a more jovial, playful, and creative Zeke.

Engineering

My internal engineering function has worked hard to deliver this site. It's build with Eleventy and Tailwind. Most of my pre-existing content has already been ported.

Content

You can expect to read articles, luke-warm takes, talks and shitposts. The Content Function is still rerecording some older talks and articles. Stay on the lookout for those. They will be backdated to their original release dates.

A word of caution on hiring rubrics

2021-11-01T00:00:00Z

Engineers and businesspeople don’t always see eye to eye. One of the few things we truly have in common is an affinity for (or maybe an obsession with) the quantifiable. It’s no surprise, then, that many hiring processes are driven by rubrics. I’ve come to believe that—for all their benefits—hiring rubrics hurt organizations more than they help. This is especially true for small organizations.

Hiring is scary. You have to make a yes/no decision about a person with only squishy, intangible information to go off of. False positives are expensive and profoundly painful. False negatives are less painful but sometimes just as expensive because of the opportunity cost of losing someone truly great^[1].

In almost all cases, a group of people contribute to a hiring process. There are good and bad reasons for this. On the plus side, a successful candidate will have to work with a diverse group of people who serve in various roles. Getting a wide sampling of perspectives is undoubtedly a good idea: it increases the likelihood that someone will identify a true red flag or pick up on a crucial subtlety. Another reason many organizations put hiring decisions in the hands of committees is quite poor: no one wants to take the heat if a hire goes bad. It’s too scary for any individual to take the decision head-on, so the hot potato gets passed around the table.

Sometimes committees find consensus; in many cases they don’t. This is where rubrics become important. Everyone fills out a survey about the interview, often with 1-5 as the options. Each candidate gets an aggregate score. This score can be compared to other candidates and may steer the committee towards a decision or even determine the outcome outright.

Rubrics may make the process easier, but I’ve come to believe that they lead to worse outcomes^[2]. People feel better about rubric-driven processes because they mistake precision for accuracy^[3]. When someone sees a number, it feels like it comes from some kind of scientific instrument. But it doesn’t. At best, it’s a numerical representation of someone’s deliberate reflection, and at worst it’s a hot take.

When rubrics are used at a company, someone, someday will be silently thinking, “Well, I really thought [candidate] would do great here, but I can’t argue with the fact that [candidate] scored a 6.2 on [topic]… I guess that’s not a good enough score.” All nuance is swept under the rug, everyone’s hands are clean, and no hard conversations are had. No one has to go out on a limb.

Proponents of rubrics claim that they’re necessary for insuring consistency and fairness across a company. Without a rubric, they ask, how can you make sure you’re rooting out unconscious bias? At a certain scale, rubrics might be helpful in ensuring organizations operate consistently (although I think it’s pretty clear that this is often an illusion). But, I’m not so sure rubrics reduce bias—conscious or unconscious^[4]. In fact, I think it might make the problem worse because it lets bias hide behind a number. The world is awash with examples of bias codified in rubrics or algorithms. For example, the SAT is widely understood to be discriminatory.

Proponents will ask how they can compare two candidates without a rubric. I would respond by asking them how often this is truly a problem they have. I’m sure there are some cases where companies run highly concurrent interview processes. But, in most cases, candidates aren't on the market at the same time. They come through as a stream and have a short time window in which you can make the hire. You might have to compare a candidate to 1 or 2 others, but the rest of the application pool has expired. Do you really need numerical data to make that kind of decision?

So what do I propose instead? Well, honestly, this is hard stuff. If I had a perfect answer, I’d build a perfect company, get rich, and retire to build wind farms.

I think the best answer I can come up with is one that embraces the hard part. Make someone go out on a limb. To be hired, you need someone to have conviction. Anyone can raise a red flag.

Have a committee interview the candidate. Make sure you have representation across various races, genders, roles, and experience levels. Consider excluding very recent hires who might not know the company dynamics well, yet.
have interviewers fill out a short qualitative survey. The survey should be a thinking tool. Make sure they fill it out before talking to other interviewers. You don’t want groupthink.
Everyone meets and reviews the surveys (anonymously)
If anyone raised any true red flags^[5] in the surveys, the group discusses, but the candidate is rejected in all likelihood. The bar for dismissing a red flag is extremely high.
The group discusses the candidate in an open fashion.
Does anyone on the committee have conviction that this is a great hire^[6]? If not, reject the candidate. It’s very important to create space for different styles of conviction to emerge. Not everyone communicates in the same way.
If you really want to use numbers, try expressing conviction as a bet... "how much would you bet that [candidate] is able to succeed in this role." That allows people to quantify confidence and risk sensitivity.

What I like about this method is that it’s sensitive to disqualifying information, but also to positive conviction from an individual. All it takes to be rejected is a red flag. To be hired you need to have someone believe you’ll be amazing.

Have I tried this method with great success? No. This is what I’m going to try next time I build a hiring process. So, take my advice with a grain of salt. My thoughts on this are fluid. And, even if I had used this method with great success, that doesn’t mean it would work for you. Hiring processes reflect culture, and every culture is different.

One final piece of advice—and this I write without any caveat—If you’re part of a hiring process that uses a rubric, and you don’t think it’s serving the company or the candidate well, you can do something about it. I’d suggest you voice conviction in spite of what the rubric says. Don’t sit quietly and let your conviction fade. That’s not to say you shouldn’t allow yourself to be influenced by your coworkers perspectives… you should. But don’t allow yourself to be silenced by a random number. You could say “I know the rubric has [candidate] as a 5.3 on [topic], but are we confusing precision with accuracy? I got a really good feeling about [candidate]’s ability to take on this role. I think we might have a misread on our hands.”

I’d love to hear from you. Have you had experiences with rubrics—good or bad—that you think I’d benefit from hearing about? Let me know. Have you found a system that you believe in? I’d love to hear about that too.

This is an example of Loss Aversion ↩︎
I don’t have any hard evidence for this, but based on my personal, subjective experience, I believe it ↩︎
This is an example of Precision Bias ↩︎
It’s possible that rubrics could help if interviewers were blinded from identifying information about candidates, but this is in all likelihood impractical. Would that even be an interview? ↩︎
What’s a true red flag? That’s up to you to decide, but, in my opinion it’s discriminatory behavior, ethical concerns, incongruence with company values, etc. Red flags can sometimes be subtle, but they’re always about something serious, otherwise they wouldn’t be a red flag, they’d be a concern. ↩︎
It’s important that you don’t have trigger-happy interviewers who have conviction for everyone who walks in the door. I recommend that before someone starts conducting interviews, they shadow the process for a little while. They conduct interviews like normal, but their conviction isn’t enough to make the hire, at first. ↩︎

Semantic operation layer

2021-11-02T00:00:00Z

The last couple of applications I’ve developed have eventually ended up craving a design pattern that I’m starting to call an “operation layer.” I've never fully realized it, but I think I'm onto something here. As with so many things, this may already exist in some form. If so, please tell me about it.

What

Operations are units of action in your application. Your application doesn't do much of anything outside the context of an operation. Create a new user account? That's an operation. Send a reminder email? Another operation. Scheduled script to cleanup dangling data? That's an operation, too.

Operations aren't just HTTP Requests. Jobs are operations. CLI commands are operations. Operations are a wrapper around a unit of work that could be connected to any transport layer.

Operations are code, but they're also data. They exist in the codebase, but at runtime they produce artifacts that can be used to debug, analyse, and understand your users and application. They can be associated with a user, machine, etc.

Operations can even be related to other operations. For example, if a user signs up for your service over HTTP with a CreateAccount operation, and a SendWelcomeEmail job operation is enqueued on a background process, the SendWelcomeEmail operation has a parentId referencing SendWelcomeEmail.

Why

Semantics. Operations are a semantic system that's decoupled from your API or transport layer. It's almost your private public API.
Configuration. Operations can have all sorts of declarative metadata (whatever your application calls for... permissions, log levels, database transaction rules) that specify how the operation should work. Depending on your environment, they could be lined up 1:1 with input validation (in the ts world, maybe have 1 class-validator class per operation?).

Visibility. They can be tied into your logging, APM, and and error tracking systems, so that you can trace individual events back to a user and an operation. For example, they could produce structured logging artifacts that go into Elasticsearch through stdout.
Adaptability. Today you might have a REST api in front of many operations, but you can switch that to a GraphQL API or something else down the road. By disconnecting your transport layer from your semantic layer, you aren't tied down to one technology. You could even go so far as to have Operations be a stepping stone to an event-driven architecture like Event Sourcing/CQRS, in which you persist operation data to a datastore, and have the rest of your application's state react to that data in various projected datastores.

How would this look?

I'm going to demonstrate the way I imagine this working in a node app. This is fictional code.

export const CreateAccountOperation = Operation.create({
  name: "createAccountOperation",
  input: CreateAccountInput,
  // the specific configuration options are arbitrary
  // and only intended to be illustrative
  logLevel: LogLevels.INFO,
  allowUnauthenticated: true,
  timeout: 1500,
  maxRetries: 1,
  run: async function (input) {
    const session = userService.createAccount(input);
    return session;
  },
});

If we were to expose this operation over HTTP, it would look something like this.

const createAccountController = (req, res, next) => {
  CreateAccountOperation.run(req.body)
    .then(({ jwt }) => res.status(201).json({ jwt }))
    .catch(next);
};

export default Router().post("/", createAccountController);

I imagine Operation would make use of node's new AsyncLocalStorage class to make use of Dynamic Scoping^[1] to share state to every future point in the callstack without passing it directly. In a way, it reminds me of React's Context, if you're familiar with it.

I'm imagining something resembling the code sample below. This is pared down from my example for the sake of clarity, but it's basically right.

export const operationStorage = new AsyncLocalStorage<{
  id: string;
  operationName: string;
  // perhaps userId?
}>();

type OperationConfig = {
  name: string;
  run: () => Promise<unknown>;
  // here's where you put _your_ configuration items
};

class RuntimeOperation {
  private readonly id: string = uuid();

  constructor(private readonly config: OperationConfig) {}

  public async run() {
    return operationStorage.run(
      {
        id: this.id,
        operationName: this.config.name,
      },
      async () => {
        return this.config.run();
      }
    );
  }
}

export class Operation {
  private constructor(readonly config: OperationConfig) {}
  static create(config: OperationConfig) {
    return new Operation(config);
  }

  public async run(): Promise {
    const runtimeOperation = new RuntimeOperation(this.config);
    return runtimeOperation.run();
  }
}

Logging

If you wanted a logger that would always report the operation name and runtime ID, you could wrap your existing logging library like this:

import bunyan from "bunyan";
import { operationStorage } from "./operation";
const logger = bunyan.createLogger({ name: "log" });

function makeWrappedLogger(level: "info" | "trace" | "debug" | "fatal") {
  return function (statement: Record<string, any>): void {
    const { id, operationName } = operationStorage.getStore() || {};
    logger[level]({ ...statement, id, operationName });
  };
}

export const info = makeWrappedLogger("info");
export const trace = makeWrappedLogger("trace");
export const debug = makeWrappedLogger("debug");
export const fatal = makeWrappedLogger("fatal");

If logs are being persisted somewhere searchable, this gives you a lot of power. If you know there's a problem with an operation, you can search for it by name. If you were to add a userId to your operationStorage you could also search by that.

What do you think of this pattern? Tried anything similar? What were the results? Is there a framework or language that makes this easy?

I learned about this term from my friend Omri Bernstein. Dynamic scope stands in contrast to lexical scope. To quote Omri, "Lexical scope is all about accessing named references based on code’s definition whereas dynamic scope is all about accessing named references based on code’s execution... React context and AsyncLocalStorage give you a lexical reference that you can dereference to get a dynamic reference, which prevents global clashing, and gives you back some of the traceability of lexical scope" ↩︎

Why is Excalidraw so fucking good?

2021-11-04T00:00:00Z

Disclaimer: This may read like an ad. It's not. No one's paying me to say this, I just love this product.

Excalidraw. Is. So. Good.

Why? When you first open it, it looks kind of sloppy. It almost evokes Kidpix, or something. Let's dig into what makes it 👨‍🍳💋 from a product perspective.

My threshold for trying new products is comically low. I'll try anything once. But, usually that's where it ends. I don't pride myself on being picky, it's just that most things don't find their way into my workflow.

Once I picked up Excalidraw, I haven't put it down. It became my tool-of-choice for diagrams, wireframes, brainstorms, and more. Part of why it was such a good fit for me is that it filled a massive hole in my toolkit. Before the pandemic, I would frequently sketch on whiteboards with my coworkers. In high school, I had a small classroom-sized whiteboard in my bedroom. The pandemic hit me like a ton of bricks in all sorts of ways. At work, one of the hardest parts was losing the white board. Of course, I could get a personal whiteboard, but then I wouldn't be able to really use it for work.

So what is it about this software? Why does it work so well?

You're in

There's no onboarding. No signup. No confirmation email. No OAuth. You're just in the product. Are there any flashy overlay tours with annoying animations and pulsing buttons? Is there a modal upselling you on a paid plan^[1]? No. You're just in product, able to use it.

You already know how to use it.

It's not overwhelming. There are 9 tools. The icons tell you what they do. There are no new concepts. If you don't know what something is, you can learn it in about 12 seconds.

Other tools offer quite a bit more power, but require an investment to learn. I've often wanted to try Omnigraffle after seeing what Gabe Lebec could make with it when we worked together. But, every time I've tried it, I've been overwhelmed by the learning curve (not to mention the dollar curve).

I would resort to making diagrams in Keynote, HTML/CSS (just because I know it), Graphviz, or pen and paper.

Low fidelity

It forces you to stay low fidelity. Because everything is rendered with rough.js (I think...), objects look sloppy and hand-drawn. This liberates you from the details and lets you get to the point. In a proper design tool, an incomplete thought would feel discouraging. The product would beg for polish. In Excalidraw, polish is illegal, allowing you to stay in idea-space.

Intuitive nod to pro tools

It respects standards. I've spent a lot of time learning keyboard commands for other software (Keynote, InDesign, etc), and when I reflexively use them in Excalidraw... they work^[2]! Cmd + option + g creates a group, Cmd + option + shift + g removes a group! You can hold down the space bar and drag the screen around. Too see what's possible, hit the question mark in the lower right corner.

I'm personally not a dark-mode person^[3], but some people actually won't use an app unless it looks like the dark side of the moon. Here's a diagram I recently made, rendered in dark mode.

Semantically oriented

You can build a vocabulary of objects. If you're using combinations of shapes to symbolize something, you can add them to your library of shapes, and recreate them when you need them. This is something I've wanted in Keynote for a long time.

Collaborative with zero friction

You don't have to be an Excalidraw evangelist to collaborate. Just send a link to your coworker. They don't have to sign up. They don't have to worry that a SaaS company will spam them for the rest of their life. They just click the link, and collaborate.

The takeaway

If you're building a tool and you want your users to love you and to tell their friends:

Reduce friction to zero
Make. It. Simple.
Don't upsell (or if you do, do it tastefully)
Let users collaborate just as easily
Build around users' intuition
Small building blocks can be pretty mighty.

Misc

https://twitter.com/_off_by_one/status/1362971962222534658

https://twitter.com/_off_by_one/status/1386405045684498432

They have recently taken a tiny piece of screen real estate to sell a paid plan, which is kind of annoying, have to say. But I guess they're a business? At least it's not too disruptive. ↩︎
10 years ago, one of the things I loved about jQuery was how guessable the API was. I remember being able to work on a jQuery app on an airplane (when airplanes didn't have internet) because I didn't really have to look at the docs too much, I'd just guess the function signatures. Sign of a great design. ↩︎
I decided against adding a dark mode to this site, sorry dark-mode stans. ↩︎

Helping students with pair programming

2021-11-05T00:00:00Z

I wrote an article about how to help a programming student get unstuck, in it, I offered to share my thoughts about how to help pairs of students get unstuck, and this is that article. If you haven't read the original, I'd suggest starting there, because many of the techniques that work on individuals also work on students who are pair programming.

What is pair programming

For the purposes of this post, I'm going to define pair programming somewhat narrowly. It's when two programmers are working on one computer, in collaboration, to write a program. When people pair, there are three versions of the program running--one on the computer, and two more in the minds of the programmers. If you've ever worked with distributed systems, you know that adding nodes generally makes things more complex, and this is no exception.

When individuals are debugging code, they need to synchronize their mental model of the code with the actual program on the computer. When you pair, you have to sync your mental model with the program and the mental model of another human being. In order to do that, you have to do a lot of talking. I find pair programming to be profoundly rewarding, but also fatiguing for this reason.

Does it work? Is it good for students?

Pair programming may not be for everyone. Even if you like it, you may not love pairing with everyone. As in all human affairs, personalities and styles sometimes clash.

By and large, code written by pairs is better than code written by an individual because it forces you to understand how your code will be read. Of course the code you just wrote makes sense to you--you just spent hours mind-melding with a computer. To make it run smoothly in someone else's brain, you have to also mind-meld with them.

But what about pair programming for students, specifically? There's a wide range of experiences. The average student will learn more, and have a better experience pair-programming than writing code alone. They learn how to talk about their work--which is hard to teach, they find bugs themselves as they talk through their program, and they keep each other motivated when the going gets rough.

When things go wrong

When students are pair-programming, it can go sideways for countless reasons. Sometimes it's fixable, sometimes it's not. Here are some of the causes of bad pairing experiences.

Confidence mismatch. There's a surprisingly weak relationship between competence and confidence in early programmers (and beyond?). Sometimes a pair have similar understanding of the material, and similar capabilities, but one party will have far more conviction about their ideas. This can exacerbate imposter syndrome, and lead to other problems.
Incompatible coping strategies. Student A needs a break and student B needs to dig in. Student A wants to ask for help, student B isn't ready to give up. Student A wants to know that it's OK if they don't finish the exercise, and student B believes that thinking about failure brings it on.
Misplaced authority. As you observe them work, you notice that one person takes on the role of an authority figure. Sometimes, a student will pull up the wrong social template, and act with authority, greatly limiting the partnership. It's possible for this social template to come from either or both individuals. That is to say that sometimes someone is accustomed to working for someone rather than with them.
Communication crunch. Take two normally good communicators, and then ask them to have a tactful conversation when they both really need to use the bathroom, and are changing lanes on the highway... they will no longer be good communicators. A similar phenomenon can happen with pair programming. You may find that some people unconsciously become tone deaf, inarticulate, and short tempered. This may or may not happen to everyone, and the severity varies. The effect tends to be proportional to mental strain, so as students become more comfortable with the material, this is less of a problem.
Bad vibes. This is something of a catch-all. These two individuals just don't get along. They keep misreading each other's language.

How do you know if something isn't working? Often it's quite obvious, but it can be more subtle, at least early on. Here are some of the signs of a problem.

Not pairing. Two computers, little dialog. This pair isn't pairing, they're collaborating on separate threads.
Signal imbalance. You're called over to help, and you're only hearing from one person. The other person is checked out, or steamrolled, and the person doing the talking is ignoring their partner.
Command stream. You overhear students working and their dialog sounds more like a stream of commands than a two-way collaboration. "now type new function... no parentheses here, I like to have arrow functions." A healthy pair will generally be bouncing ideas, questions, and encouragement back and forth.
Frustration meltdown. Sometimes things will get so bad that there's open hostility between students. This can happen for many reasons, but it's pretty hard to miss when it happens. It sounds like people fighting.

These problems show up in classrooms, but they're not exclusive to them. I haven't been teaching in 5 years, and I see these kinds of problems all the time amongst coworkers. And just because I understand them, it doesn't mean that I'm immune. Sometimes I even know what's happening but can't quite fix it.

Preventing problems

There's no way to prevent 100% of pairing problems. But, there are some things you can do from day 1 in a classroom to minimize them.

Set expectations & norms. A lot of frustration comes from mismanagement of expectations. Tell students to expect some friction. Tell them what is expected of them when it comes. Provide ground rules for how students are expected to treat each other.
Give them tips ahead of time. "If you're having a hard time getting your point across, try a diagram. If that doesn't work, try to ask your partner to say it back in their own words." "if you have a disagreement about direction..."
Tell them why students do pair programming. It's valuable, and there's a reason why you decided to set up your class in a certain way. They'll be more bought in once they understand that.
Adjust pairs. Over the course of the term/semester/cohort, adjust who pairs with who. You'll get a sense of individual pairs chemistry and have ideas for better matches. Matching isn't all about skill level (I love pairing with programmers of almost all levels, learn a ton every time). Skill level can come into play, but matching on skill level alone will not solve your problems. You could get really data-driven about it, and I've tried, but if your class-size is small enough, you'll learn who fits well with who. You can send our surveys, that's something I've tried with mixed results. If you do that, make sure students know that their partner won't be in trouble if they give a negative rating on the experience.

How to help

When you are called to help a pair, fight the urge to sidestep interpersonal issues. If you observe a dysfunctional dynamic, help the relationship before helping either individual, and certainly before "helping" the computer. If you parachute into a toxic dynamic, tap a few keys, and make an error go away, you're kicking the can down the road^[1]. That's not to say you must completely fix the social problem before moving on--that is very likely impossible. But, if you ignore it, tensions will compound, and it will be harder to address later.

Debugging interpersonal issues is tricky, and there are entire professions dedicated to it. There's no formula. If you expect yourself to be able to "fix" it, you're setting yourself up for disappointment. But, you can help, and over time things really can improve.

A good starting point for solving any problem is understanding it. Build your mental model of the social problem. Ask the students what's going on. Name the tension, and see what happens. With no judgement in your voice, try something like, "I'm picking up on some tension... is it fair to say you're both feeling some frustration right now? What's going on?" Then see what happens. If the students are willing to talk about how they feel, make sure to show them that you're hearing them. It's hard to predict where this goes, but sometimes it's cathartic for students to have the issue out in the open.

Depending on the level of tension, you may to take a break before returning to content. "It seems like lunch is in 15 minutes, why don't we break early and come back to this first thing after lunch." Time and nutrition can be as good a medicine as the best diagram or explanation.

When everyone's ready to turn to content, you can act as a communication bridge to address many communication problems. Slow things down. Make sure at every step of the conversation, the two individuals hear each other. Point out for students when you think there's a communication problem. Try bringing in diagrams, rephrasing, examples. You can also inject a different energy into the dynamic with your own reactions, "ooh that's a really interesting question!" Just make sure it's authentic... injecting bullshit into the dynamic probably won't help. In a similar vein, when there's a win, take time to celebrate it.

You can pull back the curtain and explain what you're doing. "I'm trying to improve our communication dynamic because I think a lot of the problems we're having stem from the fact that we're not hearing each other." You can also offer suggestions. "What if you used diagrams like we have over the last 15 minutes."

Throughout this process, you may experience strong feelings. You can get frustrated, too. Keep an eye on this. If you're getting frustrated, it's probably time for a break, a new approach, or to maybe pull in another teacher or TA. It is possible to make things worse.

If nothing works, talk to them one-on-one later. See if they're more open when apart.

Pair programing is fraught with problems, but at the same time, it's 100% worth it in a classroom setting. Students learn more about code, more about their own communication dynamics, and are more resourceful than they would be alone. But, without some hands on social support, there will be enough failure cases that you might conclude it's not worth it.

These techniques are personal. They may not work for every teacher or student, and that's OK. I hope they serve as fodder for thought and discussion. I'd love to hear from other programming teachers about your experiences with pair programming in the classroom. Has it worked for you? Been a train-wreck? What do you do when students have problems?

I'm sure there are exceptions to this. Perhaps the error itself is adding to the tension, and taking it off the table right away will make it easier to figure out the real issue. Trust your gut. ↩︎

Moving to a new job? You want a personal knowledge base

2021-11-13T00:00:00Z

Disclaimer: this is not an ad for Obsidian.

Onboarding to a new company is hard work. Keeping all the information straight is its own work stream that deserves thought and attention. Countless names, faces^[1], teams, initiatives, and acronyms fly by before noon. Believe me. I'm in the process right now. This week I landed on a system I'm pretty happy with.

You're a knowledge worker. Build a knowledge base

You need a place to put all the information you're receiving. This is some of the stuff I'm talking about:

People
Teams
Repositories
Relationships between teams, repositories, people, etc
Initiatives
Your workproduct and tasks
"Playbooks" for common tasks
Daily agendas
Meeting notes
Ideas
Questions you need to ask someone

I recommend Obsidian. It's essentially a markdown editor--but it's so much more than that. It's open source. It works off of your local filesystem--no cloud-based privacy concerns. It encourages you to create a dense knowledge network with many links and tags. It also has a rich set of official and community plugins.

Plant seeds

It can be quite intimidating to start from an empty directory. But, you don't have to build your knowledge base manually. Try seeding from available data sources.

Eng wiki

Does your organization have an engineering wiki on github? Clone it and dump it right into your Obsidian folder. Now you have tons of pages instead of a blank slate.

You are your company's foremost export on onboarding. No one else remembers what it was like to onboard, and--especially if they onboarded a long time ago--the experience could be quite different, now. You can use your vantage point to make the next hire's onboarding experience better. Contribute to the wiki as you're onboarding. Make it less confusing.

I recommend checking out a branch of the repo so that any changes you make are PR-able. One really easy change that adds a ton of value is just adding hyperlinks. There's often an obscure page describing something that doesn't have any links going to it. Hyperlinks are a better discovery tool than grep, so add them.

Github repos

Add a markdown file in your knowledge base for every repo at your org. If you use Github, download the Github cli. From there you can generate a json file of the top n repos at your org... if you can, maybe just get all of them. They are ordered by most recent commit.

$ gh repo list [yourorgname] --limit 1000 --json name,description,createdAt,updatedAt`) > your-repos.json

I wrote a little node script to generate markdown files

const { writeFileSync } = require('fs');
const repos = require('./your-repos.json');

const pathToObsidian = '...';

function renderMarkdown(repo) {
	return `# [${repo.name}](https://github.com/yourorg/${repo.name})
	
  >> whatever you want here, description is nice...<<`;
}

for (const repo of repos) {
	writeFileSync(
	  `${pathToObsidian}/repos/${repo.name}.md`, 
	  renderMarkdown(repo)
  );
}

People & Teams

This one is harder to make generic. At my new job, I found a pretty good list of the engineering org in Google Groups, copied the html table from the browser into Google sheets, then exported a CSV. Your source of truth might be Slack, Github Teams, or something totally different. You can also add people manually.

Version tracking

Use git. Git init in your Obsidian directory. We use git for everything else, why not our knoledge base? Commit at the end of every day and take a look at the diff. Write a real commit message. Don't worry about length. For me, it contains a better summary of my day than I can find anywhere else. $ git log is now a pretty great way to see what you've done in a week.

Just make sure you have a .gitignore file. In my case I ignore my eng-wiki directory so I don't have to deal with git submodules.

Diagrams

A picture is worth 1000 words.

Excalidraw

There's an Excalidraw plugin for Obsidian (thanks @zsviczian!). I've already written about how much I love Excalidraw. The fact that Obsidian can elegantly integrate with Excalidraw is amazing. When I say elegantly, I mean it. You can link from inside diagrams to pages inside your knowledge base. You can embed. It's great.

Mermaid

Excalidraw is a great general purpose diagraming tool. But, for highly structured diagrams, Mermaid is sometimes better. Obsidian has native support for Mermaid diagrams. Just make a code block annotated as mermaid.

Here's an example from their docs.

flowchart TD
  A[Start] --> B{Is it?};
  B -- Yes --> C[OK];
  C --> D[Rethink];
  D --> B;
  B -- No ----> E[End];

Questions

1,000,000 questions will come up every day. There won't always be time to ask them. Some of them you can look up. I recommend making a system for these. I made a central questions page. When things come up on other pages, I link to my questions page. That way, when I want to review open questions, I can go look at all the backlinks. Tags also work.

Every day take a look at open questions. Look through the company wiki, or other sources of truth. If you find the answer, update the link from [[question]] to [[answered-question]] and include the answer with any hyperlink to existing content in the knowledge base. For example, [[answered-question]] what's the protocol for opening PRs into other teams' repos? Answer: [[eng-wiki/code-review-process]] goes into this.

For questions you can't answer for yourself, filter them to only those which matter and your team can answer, and slack them en mass, each day. Make sure the answers make their way back into your knowledge base.

Daily notes

Obsidian has a daily notes plugin that is very useful. First thing in the morning, I create a daily note, enter my google calendar agenda (hyperlinking to everyone I'm meeting with). I always include a meta section for observations about how the system is serving me. I also recommend creating links referencing the next and previous day.

I find that daily notes end up accumulating a ton of random stuff that came up that day. Take a little time to ask yourself if content on the daily note page should be relocated.

Tickets

At least early on, every task you're assigned in your product management could also be tracked in your own personal note system. It allows you to link to relevant docs, take notes, draft status updates, gather questions, make diagrams, etc.

My tickets directory isn't flat. Mine looks like this:

tickets
├── closed
│   ├── POP-696.md
│   └── POP-698.md
└── open
    └── POP-700.md

Having the distinction between open and closed helps at standup when you want to tell your team how the ticket is going.

Many of the tasks you complete when you're new on the job are somewhat rote. You're not re-architecting the home page in week one. For these repeatable tasks, is there good documentation about how to do it? Probably not. Make a playbook. It'll make it easier to do next time. If your playbook describes something beyond just you and your team, add it to the eng-wide wiki.

Retros, free writing, and synthesis

If your team does retros, I recommend calendaring some time beforehand to do free writing. Even if your team doesn't have a retro tradition, scheduling some time for free reflection is a great idea--it encourages synthesis, connections, and understanding.

Think of it as part of your job description to minimize entropy. After you understand the ball of yarn that is your organization, you can untangle it, or at least reduce the rate at which it tangles. You can come up with analogies, diagrams, connections--anything that makes the system fit into brains better--you will add tremendous value. Here's a great article about this general idea as it applies to researchers. They call it research debt. I think of it as semantic debt or cognitive debt. Reflective writing is a good way to prepare your brain for synthesis.

When you write reflectively. Don't worry about formatting and linking while you free write, it could get in the way of flow. Once you're done, see if there are links to add or formatting to apply

It doesn't help that I have prosopagnosia! ↩︎

I returned my Remarkable2, here's why

2021-11-28T00:00:00Z

I'm going through one of those moments in life where everything changes at once. As such, I've been deliberately trying to branch out into new tools and new ways of thinking. A month-or-so ago, I bought a Remarkable tablet.

At this point in the article, everyone who's known me for a long time is in shock. I have comically bad handwriting. So bad that it has a diagnosis--dysgraphia.

I love typing notes. Most of my notes, and all of my writing is digital. I refuse to write code by hand unless I'm forced to in an interview. But some things are not easily typed: organic molecules, ERD diagrams, flow-charts, doodles... the list goes on. That's exactly why pen-and-paper has historically had a place in my toolkit despite its disadvantages.

Paper also has a magical quality that digital tools lack. It captures the "air in the room." When you review paper notes you can see how an agenda turns into an idea, turns into a meeting, turned into a doodle, turned into a diagram. You see how thoughts and events bleed together.

One of the things that excited me greatly about getting a Remarkable tablet was that it might combine what I love about paper and what I love about computers. Like paper, I thought it would be easy to pick up and use. Like digital products, I it would let me stay organized and hyperlinked.

I went for it

A few weeks before starting my new job, I decided now was the time to try Remarkable. It took a little while to arrive, but when it did, I was excited. The unboxing was a good experience. You can tell they spent a lot of energy on it (honestly probably too much considering its other deficits). It had an Apple feel, and even smelled like a new mac.

When I turned it on, I had one of my first disappointments. It was really hard to to type my wifi password. I have a fairly long password, and it must have taken me 5 tries to get it right. This speaks to the fact that Remarkable didn't invest too much in the typing experience. That makes sense... after all, they are building an e-paper company. But this problem exemplified something deeper. The non-writing features that you need in order to use the product are flat-out bad.

Remarkable is harder than paper-and-pencil in some really crucial ways. You'd think that it would make up for its weaknesses with digital benefits that paper can't provide. But it doesn't. You'd be better off with a paper notebook and a scanner.

It's too small. Flipping the page is way too hard.

When I finished onboarding, I decided to start a daily notes folder and make a notebook for each day. I filled up the first page quite quickly just drawing random stuff about the day. I was surprised how quickly the page filled up. It felt closer to writing on a large napkin than a piece of paper. I felt cramped. I looked it up, and Remarkable has the height and width of A5 paper.

I should note that in theory there is a gesture I saw referenced in the onboarding materials to go to the "next" page, but I never could figure out how to make it happen, and I'm not sure if it is supposed to add a page to the end of a notebook or just navigate between existing pages. When I wanted to "turn the page," I was shocked at how hard it was. I had to give the sidebar menu a colonoscopy to find the UI to list all pages, then create a new one, then navigate to it. This gap in effort between my expectations and my experience was massive.

The eraser is kind of annoying

I was relatively happy with the flexibility of the writing settings. You could write with different styles and widths. Writing felt good. Erasing was simple enough, but I did notice a subtle imprecision. The content you erased wasn't quite where you put your eraser. It wasn't a deal breaker, but just annoying enough to note.

Shitty file system

The filesystem they chose--folders and notebooks--are just not the best choice. Remarkable decided to use UI patterns from other tablets and computers but didn't mirror the UX quality that I've come to expect. As I previously mentioned, their tablet-like UI is hard to use. It just takes too much effort to move things around and get organized.

No context or tie-ins

You can't hook remarkable notes into anything. I wanted to use Remarkable in meetings. I figured there would be a way to automatically create notebooks for meetings and save that association to my calendar. That wasn't possible. As far as I could see there is no official API support either, so it's not possible to make a plugin.

I wanted to have drop tags in notebooks--impossible.

I figured that at least there'd be some way to drop a breadcrumb on the page to reference something like the current time. Perhaps I could use that at the end of the day to cross-reference meetings with notes. This was also not possible.

Bad sync

The sync feature between apps is just buggy. The setup is relatively slick, but notebooks get stuck in a stale state. I often needed to restart my Remarkable, my mac app, or both to get the latest notes to sync. There's no sync or refresh button.

You can't really read books

Remarkable is advertised as an E-Ink reader as well as a writing device. In reality, this only works if you have e-pub books on your hard drive. Where can you get e-pub books? Basically nowhere. There are many public-domain titles, but if someone were to purchase the Remarkable thinking they were also getting a kindle, they'd be very disapointed. It's true that you can annotate PDFs on the remarkable. But, that wasn't something I had a particular itch to do. Perhaps if I was already being asked for feedback on PDFs in my regular workflow, that feature would have resonated with me differently.

Comically bad OCR; No search

Remarkable advertises that they can convert your handwritten notes into text. This doesn't really work. Firstly, it doesn't happen automatically, you have to do trigger the OCR yourself. When you trigger it, it's done per-page, not per notebook, which is kind of stupid. The text also isn't saved in any notebook hierarchy, it too is page-by-page.

The OCR quality is comically bad. It picks up a few words here and there, but it just isn't what you wrote. If it did work, there's no way to search all your notebooks for text and go to the right page.

Sum it up

Let's take a moment to consider Remarkable vs pen, paper & scanner.

	Remarkable	Pen, paper & scanner
Cost	~~$399~~ $299	Free (phone) or ~$200
Monthly subscription cost	$5-8/m	$0
Battery life	~14hrs	∞
Shipping time	Weeks	Irrelevant
Page flipping	Annoying	Trivial
Size	5-7/8 x 8-1/4 in	8-1/2 x 11 in
Organizational scheme	Use clunky UI to move things	Not great. Either have binders or move scanned PDFs around
Eraser	Bad	Bad
If you lose the pen	$50-$130!	Ask a friend? Buy a dozen at Staples?

I want to say that I don't intend to knock the Remarkable team. It's clear how much thought and energy and love they've poured into their work. It just doesn't work or even make sense to me. What am I missing? Clearly something. I've met multiple people who swear by this product. So it's doing something right! I'd love to understand what makes Remarkable work when it works.

Work in progress: hookreducer.com

2021-12-19T00:00:00Z

A dev tool that lets you setup an endpoint to accept webhooks super easily. It's an order of magnitude easier to get up and running, and covers edge cases so you don't have to worry about it. Additionally, you can capture a little piece of state with a JS reducer function. Think Webhook + Redux (if that analogy makes any sense to you).

I've been a little quiet lately. In my hobby-time, I've been working on a dev-tool micro-business. I think now is a good time to share it with the world! To be clear–this is half-baked stuff. I'm sharing it because I believe that good things can happen when you work in public, and I'm interested in seeing folks' reactions. I also like leaving little time capsules of projects. I always wish I had done this more in the past. So if nothing else, this post will do that.

The idea: hookreducer.com

(Logo courtesy of the Be Nice Comic)

There are a lot of places to run code on the internet. This product sets itself apart from Glitch, RunKit, CodeSandbox, etc not just because of what it can do, but also because of what it can't. You can't run any async code. You can't access the network. You can't import any npm packages. What you get instead are special webhook-specific features. It handles idempotency tokens, request authenticity verifications, queueing inbound requests, and more.

What makes this super awesome is that when you change the reducer code, you can flow all your historical events through your reducer again to recreate the state. You can even set up an endpoint and leave the reducer function blank! Tell your webhook provider to send over data, and write your reducer code later!

User Interface Elements

Resource Bar

The resource bar is a top level component. It gives you at-a-glance access to important information about how your hook is running, but also how your hook sits relative to other hooks. Clicking on it will likely bring up some kind of modal/popover interface that gives you more access to some of these concepts.

IDE

You write code that runs every time a request is received. You have the opportunity to define a set of functions to configure the hook's behavior, including reducer, getIdempotencyKey and isAuthentic. There are several APIs available to you. The IDE has intellisense to help you explore your options, but you can also read (yet to be written) docs. Anything you console.log will be visible in the Requests table.

Requests Table

This interface is inspired by the Network tab in Chrome Dev Tools. It lets you explore the requests your endpoint. There are a couple of notable features.

You will be able to expand a request and see a diff of the state
You can see an at-a-glance visual representation of the request body. I use locality-sensitive hashing and Jdenticon to make similar payloads appear similar.
I want to build a custom-columns feature that allows you to surface parts of the request body or state, perhaps using a jq expression.

Features

Set up as many endpoints as you want and map them to reducer functions to retain state
Queue up inbound traffic to handle spikes. No race conditions internally to HookReducer.
Easily handle idempotency keys without writing any code
Change your reducer function and recreate your state with the new code.
Tag requests?
Publish state
- on a websocket
- to SQS?
- to another webhook address?
  - Maybe have the option to batch or debounce to that address? Have HookReducer handle the traffic spikes.
- to Slack?
Leave at any time: you can export your code to a zip folder that will give you everything you need to run your endpoint on your own. You can also export all your event data
Create multiple, revokable read and write keys^[1]
Write keys (endpoints) can map to multiple reducers?

Pricing & accounts

You can create an endpoint for free without even making an account
- If you don't create an account, your data end endpoint will be deleted after some period of time (2h? 2d?)
With a free account, you can do a lot. Not sure exactly which features I will gate, but I'm looking at making the free tier actually useful, not just a taste of the real experience
Paid accounts ($5-$50/month... wide range, I know, I'll figure it out later) will offer higher execution limits, more features, isolated runtimes, support, etc. The idea is that if this has really made it into your workflow, and you're seeing lots of traffic, you need to pay for it.

This solves a non-trivial problem

Webhooks are ubiquitous. Server-server notifications come up any time you're building an internet product. For example, if you're building a SaaS product, you might need a provider like Stripe to send you a webhook any time someone's card is declined because you need to send them an email.

Building an endpoint designed to accept webhooks isn't rocket science. But, there are some gotchas, and doing it well is non-trivial.

Idempotency

You can't count on a webhook provider to send an event once and only once. If your system receives the same event twice, you might send the customer an email twice.

Spikes in requests

Sometimes you get unpredictable spikes in traffic from webhook providers. This can be for a variety of reasons^[2]. It's important that when you accept webhooks, you queue incoming requests to be processed asynchronously. That way your webhook doesn't take down production. HookReducer will be built to accept requests at scale and process them later. Everyone goes down, but the architecture is built to minimize downtime of endpoints.

Authenticity

How do you know if the request your endpoint is receiving is authentic? Anyone who knows the address of your endpoint could hit it. This is a kind of "monster-in-the-middle attack". Now, you could share a secret with your provider that only they know. But that isn't too different from an obscure URL. The way some services (e.g. Stripe, but many others) handle this is with an authenticity token. The token is often an HMAC hash of the payload. If you know a shared secret that the provider shares with you, you can use it to hash the payload and compare it to the authenticity token sent. This isn't the way to authenticate webhooks, it's a way. But, the point is that Hook Reducer will give you the tools you need to authenticate hooks from a variety of sources.

My goals

My primary goal is to make something people care about. For this to work, it has to pay for it's own hard costs, and if I can make a little money on top for my time, that's great. This is explicitly not a "startup." I don't believe I'll ever raise money, hire employees, launch a marketing campaign. Once feature-complete, I'd like to be able to run this thing with about 4hrs/week (with the exception of new product development). I work fulltime and this isn't intended to replace that.

I also have a chronic tendency to abandon side projects. My hope is that by working in public I'll be able to see this thing to it's natural conclusion!

What I'm looking for from you

I want to know if people are excited about this? Do I get old friends coming out of the woodwork to tell me they'd use it? People telling me about feature ideas? Crickets? I'm trying to take the pulse of the internet and see how you all feel. Have some advice for me? It is hereby solicited.

If you want to remove someone's ability to see the state or change it, for example ↩︎
Examples: their queue got messed up after some downtime and they send you 5 hours of activity all at once, someone bulk updates customer records and they send you a request for each one, etc, etc ↩︎

Micro-business update: name change, software coming together

2021-12-25T00:00:00Z

Hello! It's been about a week since I first wrote about working on ~~Hook~~Web Reducer. I'm on vacation from work and have some time to invest in this, and a lot has changed. I'm getting closer and closer to shipping an MVP. That's the next milestone. The vision is also sharpening.

Business updates

Name change

Something about HookReducer wasn't sitting quite right with me. I loved that I got the .com, and what I'm building is well suited to webhooks. But it's not just for webhooks. It's really for any kind of message. Initially, I'm only accepting messages via HTTP (and it could stay that way forever). But the client doesn't have to be a webhook provider.

WebReducer.dev struck me as a better name. It's still easy to spell and say. I think it's memorable. The .dev TLD is generally less desirable than .com, but in this case, it immediately tells you that the product is a service for software development. HookReducer.com could be selling something to hang curtains, a sewing tool, etc, etc.

Open source

I've decided to open source this project now, even though it's nowhere near feature-complete. I always intended to build an open product. I imagined features that would allow users to leave anytime they want, with their data and their code, and run on their own infrastructure. I was also leaning towards open sourcing everything. Once I made this decision to do it, I wondered when was the right time. I was a little bit fearful that someone would look at my code and judge me. Why did I use x-library? Why don't I use Y Design pattern there?

I've reflected a little bit and gotten over these fears

Criticism can be very useful, so why not have it
When it isn't useful, ignore it
If I let these fears prevent me from open sourcing, I won't really be building in public, which was part of the whole point of this project. I believe building in public is partly about letting people see the process, not just a polished product.

So, I've made the repo public as of now.

Product

I've made some progress on value prop. I think I can articulate some use cases more clearly than I could last week. A point of emphasis: this is a µ-product. It's designed for µ-problems, of which there are many. Here are some I've thought of.

Prototyping
- Often times when you're working on a new project, it's somewhere between prototype and production application. Maybe you don't have as many tests as you normally would. Maybe you don't have as much observability as you might want. Maybe you don't handle webhooks all that well. Start with WebReducer. Put it in front of your project.
- Maybe you have a new integration in your product. It's immature, and you're not sure if it'll stick around. They have a webhook feature, but you don't want to do the legwork to setup a queue, validate authenticity, etc, etc, etc. Put WR in front of your app. Write your reducer function later if you ever need it.
- Your building a client-side prototype and don't want to bother with a real backend, but would love to save a little data somewhere. We've got you covered
IOT
- Do you have devices out in the world that are intermittently connected? Maybe you want them to be able to connect to a server and ask "what happened since I last had a connection." WR is perfect for that.
Business observability
- Perhaps there are business events you care about on a 3d party platform. These events matter to the business, but not to the software. It would take engineering resources to create webhook endpoints for them. You just want to stick them somewhere and do something with them later.
Games
- You want multiple players to agree on the state of the world
- Multiple players can send in events
- All players are notified via websocket when the state of the world changes
Personal
- I want to expose a read endpoint that tells my friends how many unread messages are in my gmail inbox using the gmail api

Engineering

On the engineering front, I've made the most progress. I've added many security measures for the code runner.

High level architecture, security, and resilience

The system is designed with two priorities

Go down as little as possible
Don't let user-generated code do anything bad

To achieve this, I make heavy use of queueing and isolate the hell out of the VM that runs user-generated code. I took some inspiration from Tim Nolet's post.

flowchart TB
    Client[Client React App]
    HookClient[Endpoint Clients]
    Server
    subgraph isolated-docker
    Runner
    end
    Worker

    Redis[(Redis)]
    Postgres[(Postgres)]

    Client <--> Server
    HookClient <--> Server
    Server <--> Redis
    Server <--> Postgres
    Worker <--> Redis
    Worker <--> Postgres
    Worker <--> Runner

When requests hit WebReducer, they are put directly into a bull queue. The only thing that happens before that is body parsing. If we see a surge in traffic, the database won't be the point of failure. These enqueued jobs are processed with a reasonably high concurrency. The initial processing doesn't run the code, it just captures the request in postgres and enqueues the next job to run the code.

%%{init: {"flowchart": { "useMaxWidth": false } }}%%
sequenceDiagram
    participant Client
    participant Server
    participant Redis
    participant IngestRequestWorker as Injest Request Worker<br>( Concurency ⬆️ )
    participant Postgres
    participant RunnerWorker as Runner Worker
    participant Runner

    Client ->> Server: Sending request
    Server ->> Redis: Enqueue job for request
    Redis ->> Server: Enqueued
    Server ->> Client: 202<br>(btw, the req id is...)
    Redis ->> IngestRequestWorker: Job
    IngestRequestWorker ->> Postgres: Save this request<br>in a persistent way
    Postgres ->> IngestRequestWorker: ��
    IngestRequestWorker ->> Redis: Enqueue a job to<br>actually process<br>this request
    Redis ->> IngestRequestWorker: I got u
    IngestRequestWorker ->> Redis: Finished w/<br> the job btw
    Redis ->> RunnerWorker: Job
    RunnerWorker ->> Postgres: Gimme that req data
    Postgres ->> RunnerWorker: {...}
    RunnerWorker ->> Runner: Hey can you run<br>this code with <br>this payload?
    Runner ->> RunnerWorker: For sure
    RunnerWorker ->> Postgres: Save the new state plz
    Postgres ->> RunnerWorker: Yup
    RunnerWorker ->> Redis: I'm done

The launcher queue has a concurrency of 1. It's very important that two launchers are not run at the same time because they would run into a race condition. Whoever saves the state last would win. You might note that a concurrency of 1 is not scalable. To solve this, I've created a series of N launcher queues. When a job is enqueued, it is assigned to one of them by a hash of endpoint key. Individual endpoints process requests serially, but the system processes requests in parallel.

The runner itself is stateless. I can create as many instances as I want if it ever becomes the bottleneck.

flowchart LR
    server[Server]
    capture[Capture Request Job]

    hash{Hash}

    subgraph runner-launchers[Runner Launchers - Concurency 1 for each]
        direction TB
        L1[Launcher-1 Queue]
        L2[Launcher-2 Queue]
        L3[Launcher-3 Queue]
        L4[Launcher-4 Queue]
        LN[Launcher-n Queue]
    end

    runner[Runner]

    server --> capture
    capture --> hash
    hash --> L1
    hash --> L2
    hash --> L3
    hash --> L4
    hash --> LN

    L1 --> runner
    L2 --> runner
    L3 --> runner
    L4 --> runner
    LN --> runner

Recomputing state

One of the best features of WebReducer is going to be recomputing your state when you change the code. But, if you've received 1M events, it doesn't make sense to enqueue 1M jobs in the queue and hit the runner 1M times. I've built the system to be capable of batching requests. This could come in handy at some point for high-volume endpoints, but for now, I'm only conceiving of it for replaying state.

bulk update service

UI

Sticky date line

I realized that in my initial design prototype, there was a whole column in the requests table that was often displaying the exact same data:

So I added a line in the table representing the date change. Here's how I made the date line. Whoever thought of position: sticky deserves a medal.

Modals

I realized that modals are going to be an important UI element, so I setup a system I'm pretty happy with.

When you want to open an existing component in a react component, here's how you would do that

const YourButton = () => {
  const { openModal } = useModals();

  const openFooModal = useCallback(
    (fooName: string) => {
      openModal({
        name: "foo",
        props: { fooName },
      });
    },
    [openModal]
  );

  return <Button onClick={openModal}>Open Foo Modal</Button>;
};

To create a new modal:

import { registerModal } from "../modals/ModalProvider";
import {
  CenteredModalContainer,
  ModalBody,
  ModalFooter,
  ModalHeader,
} from "../modals/Content";

type MODAL_NAME = "test";
const MODAL_NAME: MODAL_NAME = "test";

declare global {
  namespace Modal {
    interface ModalTypes {
      [MODAL_NAME]: {
        name: MODAL_NAME;
        props: Props;
      };
    }
  }
}

type Props = { text: string };

const TestModal = ({ text }: Props) => {
  return (
    <CenteredModalContainer>
      <ModalHeader title="Are you sure you really want to" />
      <ModalBody>foobar! {text}</ModalBody>
      <ModalFooter>Footer goes here</ModalFooter>
    </CenteredModalContainer>
  );
};

registerModal<MODAL_NAME>(MODAL_NAME, TestModal);

Features of this system:

Typed in modal definition and in openModal
Modals are a global concern like URL navigation, so it's good to have them at the top of the application
Modals are at the top of the tree, not sprinkled everywhere
Modals can be launched from anywhere in the tree

Commits:

Things to consider before joining a bootcamp

2021-12-28T00:00:00Z

Bootcamps, bootcamps, bootcamps... They're everywhere, and everyone has an opinion.

On one end of the spectrum are the hard-skeptics. The hard-skeptics believe that bootcamps are selling snake oil–it's impossible to teach what they claim in 3 months. If you went to a bootcamp you were a victim.

On the other end of the spectrum are the evangelists. In their view, the fact that software engineering bootcamps work proves one of their priors: traditional education is broken, bloated, and corrupt. School has a lot to learn from bootcamps.

If you're only exposed to strong opinions, it can be disorienting. It's challenging to know what to believe. It doesn't help you want to believe in bootcamps and there are pushy sales people feeding into your desires.

I consider myself relatively well informed, if a touch out of date. I was the first instructional hire at an early, highly successful bootcamp. I helped write curriculum. I conducted admissions interviews. I even opened our second campus in Chicago. I firmly believe that we were doing good, interesting things. The vast vast majority of our students got more than we promised. I brought in friends and family as students and staff.

It worked, basically. That's not to say it was flawless. There were problems. But, at the end of the day, students were hired as software engineers and are today enjoying successful careers^[1].

On the other hand... there is absolutely snake oil on the market, and if you or someone you know is considering attending a bootcamp you should tread carefully. Just because bootcamps can be executed successfully does not mean the one you are looking at is. Just because a bootcamp was terrific 5 years ago doesn't mean it is today. If you're like most people, the tuition is a non-trivial amount of money, and you could absolutely leave with nothing to show for it.

So what are you to do? I hope to offer some guidance.

Good bootcamps have real admissions processes

One of the biggest red flags to look out for is when there's no admissions process or when admissions feels more like a sales pitch than an interview. Admissions exists for a reason. Selectively for its own sake is gross, but, selectivity can have a vitally important function: determining if the student is likely to be successful in the program. You want them to be cautious about taking your money.

There are all kinds of reasons why a student might not be successful, the biggest of which is that they don't enjoy programming. Of those students who do enjoy programming, some may not have the space in their life to dedicate to an immersive program. Others might struggle with the format of bootcamps. The list could go on.

While I believe that everyone has a right to be exposed to technology, and that not enough people have access to tech education, it's just not the case that you could plop a random person in a room for 3 months and turn them into an entry-level software engineer. There is variance in the human experience, and a lot of it. The amount of time it takes to go from first line of code to professional engineer varies. The experience level of the applicant pool varies more.

There's a lot of BS gatekeeping in tech. I believe that the vast majority of people who like programming can get jobs doing it. But, the path looks different for everyone, and there are a lot of people who aren't set up to succeed at a bootcamp.

Good bootcamps screen out students who are not likely to be successful for the sake of the student. Good bootcamps take the attitude, "we're not going to take your money and your time if we don't think you are going to be successful after our program, and in order to know that, we have to spend some serious time talking to you."

If a program does little to no screening, that can mean one of two things. Maybe they have developed a curriculum so good that it can support students from all walks of life and get them all on a level playing field in 3 months... Or, maybe it's snake oil.

(A big caveat to this... if a program isn't making any claims about the job market or outcomes, then maybe this isn't true. Theoretically, there could be a school just focused on exposing people to programming and adding value in a set period of time. I think this type of thing would be fantastic for kids.)

Good bootcamps are serious about metrics

What percentage of students are hired after 6 months? When was that data sampled? 2017?? Why?

Be skeptical of data and ask questions. Good bootcamps treat data gathering a serious task, not a marketing activity.

Good bootcamps hire some of their own, but not too many

I've seen some bootcamp skeptics say it's a bad sign when bootcamps hire former students and TAs. Before I worked at a bootcamp, I might have also been somewhat skeptical. But, I've seen it work really, really well. We had a smart, talented and energetic student body. They knew our curriculum, and we knew them. Why wouldn't we hire them? Instructor-alums also remember what it's like to not understand something. That can make them very effective teaching partners.

I worked with instructor-alums who made tremendous contributions to our curriculum (in some cases far more consequential than my own), were dynamic and effective in the classroom and were extremely well liked by students. It's not a red flag if a school hires former students. I actually think it says something good: people went through this program and didn't want to leave.

But, I think there is a precarious balance to strike. When a school relies too heavily on recent alums for its staffing needs, things can become insular. When questions come up about how things work in the "real world" no one knows who to ask. It's a good idea to ask bootcamps about how many of their staff are alums. Make sure that some of the people teaching you have non-bootcamp software engineering experience.

Good bootcamps strike a balance with curriculum updates

Tech changes really fast. The languages, libraries, and techniques you learn at a bootcamp won't be the ones you're using in 5 years, and they might not even be the ones you're using at your first job. Bootcamps can't write their curriculum once and be done. They need to keep it current enough. That said, if they're constantly rewriting it, it can be unstable. Good bootcamps strike a balance. They make curriculum updates often enough that the lessons aren't stale, but not so often that every cohort is in alpha mode.

Good bootcamps know what they aren't

Bootcamps are not a replacement for computer science programs or college. They have very different ends. Computer Science is an academic discipline. Computer Scientists are in the business of discovery. Bootcamps are entirely industry-focused. At a great bootcamp you learn to assess computational complexity and get a solid overview of data structures and algorithms, but it's not a CS degree. Sadly, many students who graduate from bad CS programs do so with less practical understanding of Computer Science than students who went to a great bootcamp. I taught several CS grads/dropouts (from pretty good undergrad programs!) who were not planning on continuing on in their academic CS career and also weren't job ready.

Bootcamps are not a panacea. They are not the solution to every problem you can think of. You can't take a random laid-off worker, stick them in a bootcamp, and expect consistently good results.

Why do bootcamps work anyway?

I've sometimes joked that our school could have been just as successful if we had no curriculum. This is an intentionally provocative statement, but I really wonder if it's true.

Most of our success can be explained by a few factors.

Our students were highly motivated. They had put their lives on hold and were all-in.
Staff was bought in. We were right in there with the students, so much so that we risked burning ourselves out.
Our students were ready for what we were teaching (see admissions above). To get in you needed to be able to recurse through several data structures in your language of choice. Day 1 wasn't "Intro to Variables" it was "Let's write our own hash table"
Students' play time wasn't rate-limited. Unlike in so many fields, they didn't need to wait for any external parties to get feedback. Students never waited for me to grade their programs to see if they worked. The computer told them it worked!
We could get them unstuck. Getting stuck is one of the biggest obstacles to learning how to code. It's hard for a student to know the difference between productive and unproductive struggle. When you spin your wheels for too long, your motivation starts to degrade. Once that happens, you're done.
The community was supportive and motivating. If you were in a rut, you could look around and see peers that were on the same journey. You weren't alone.

A lot has changed since I left in 2017, so I can't say I'm well informed enough to speak to any specific bootcamps' quality, today. That's why I'm not naming them. But I can say that they can and do sometimes work. ↩︎

Lessons learned from a recent job search

2022-01-23T00:00:00Z

Toward the end of 2021, I was one of the countless tech workers who left one job and took another. The process surprised me in a number of ways. I needed to update my priors about a few things, and by writing this, maybe I'll update yours too.

First, I thought it would be easy^[1]. My reasoning was as follows:

I know how hard it is to hire great engineers
I know I'm a good engineer..
Everyone's desperate to fill roles...
So it should be a walk in the park.

It turns out it wasn't. It wasn't radically hard, but it required significant thought and energy. In the middle of the search, I got kind of pessimistic... And yet, in the end, it was successful both in terms of the total number of offers received and their dollar values.

I've tried to capture some of my learnings in a form that may be useful to you. Take everything I say with a grain of salt. Everyone's experience is their own, and this may not apply to you. Also the TTL on any post like this is probably pretty short.

Understand that the system is overloaded

We said that 2020 was the year that broke every time-series chart and 2021 said "hold my beer." Everything's overloaded, and the job search infrastructure is no exception. In-house recruiters I spoke with had a hoarse voices, were late to meetings and occasionally confused me with other candidates. And I don't blame them. They are dealing with unprecedented pressure from their employers, while at the same time they have more applications than they can process.

It's almost like a Retry Storm. Companies are so overloaded with applicants, that end up giving them a subpar experience. Employer latency causes applicants to freak out and put out more applications, recursively exacerbating the problem.

My advice:

Take a deep breath. You'll feel compelled to draw conclusions based on incomplete information, choose not to. Silence from companies can be harder to deal with than rejection because it begs us to read into tea leaves. Just pause. Conclusions will come in time.
Recruiters are tired, give them some grace. They're trying. Some of them are new on the job themselves. Others just had their coworkers quit. Try not to take it personally.
It's ok to politely check in. As with all things, there's a right timing and dosage.
Be gracious in cases of rejection. It doesn't matter how great you are, you're not a good fit everywhere. And even when you are, misreads happen. Maybe they're making a mistake, or maybe they're saving you from one. There are more fish in the sea.

Network

I've heard it said that most jobs are filled through networking... I don't know if that's been rigorously established, but it's the kind of statement that just feels true. When I started my search, I decided that I didn't want to rely solely on my existing contacts for introductions because I didn't want to limit myself to my known universe. I wanted to branch out. Broadly speaking this was a waste of my time. Most of the companies that I applied to without a contact never wrote back to me, or wrote back so late that they were so hopelessly behind that it was pointless to consider them.

I expected networking to be effective, but I was surprised by the degree. Given how overloaded the system is, networks allow you to punch through the noise and get in front of the right people. I wouldn't disregard conventional wisdom on this one. Use the networks you have.

Learn from your cover letters

I was surprised how many cover letters I had to write, even with a warm introduction. Most companies asked me to write a cover letter. I knew it wasn't going to be read closely, but it required non-negligible attention from me. I didn't want my cover letters to stand out in a negative way^[2].

I observed a lot of variance in the effort required for me to write a cover letter. Sometimes an articulate and crisp cover letter just fell out of my mind. Other times it was like pulling teeth. I started recording the subjective effort required to write a cover letter. I realized that the ease with which I wrote a cover letter was a decent proxy for something. If I couldn't articulate in a cover letter why a company was write for me, maybe that's a sign that I'd struggle in an interview. Maybe it's a sign that I'd be hard to tell my friends and family about the company once I start. Maybe it's a bad fit.

Like with all metrics, it's imperfect! But it's something.

Take-homes

Take-home projects don't seem to be quite as ubiquitous as they were a few years ago, but I still had to do quite a few of them. Often it only takes a couple of hours to meet the bare minimum requirements, but require significantly more effort to stand out. This presents some unfortunate systemic biases. For example, if you spend a lot of time taking care of a family member, and are doing a job search while working full-time, it's extremely difficult to find the time or energy to do take-home work well.

I saw a couple of companies pay me to do a take-home assessment, which I appreciated. It was a signal that they valued my time investment.

Even though the system is pretty unfair and imperfect, I have some pointers for how you can stand out in take-homes that I think are worth sharing.

Pick something to nerd out on

Looking for a way to go above and beyond? Pick some aspect of the problem and take it to the next level.

Maybe you're being asked to write a program that reads stdin, does some processing, and writes to stdout. It'd be trivial to do the work in memory, but you could decide that you want to stream the input and output so that your program supports arbitrarily large input sizes.

Maybe you decide you want to go a step further and load test it. Perhaps you can cover an edge case in an interesting way?

I found that take-homes often fell into one of a few buckets

Data processing script
UI
Fullstack app

When you do one, you can start to borrow little flourishes from one and use them on others. The problems are different enough that you can't copy meaningful code, but you can absolutely borrow structure, testing techniques, and sparkles from one project to another.

Have a good readme

Spend more time than you think on a readme. It may seem ridiculous to document a silly sample project, but it will show that you care about the job you're applying to and demonstrate an ability to write for humans.

Talk about what you did, how to run it and what went into your design decisions. If you're being asked to build a UI, include screenshots–many of the people reviewing your code won't have the time to fire it up. For non-ui take-homes, include sample output. The point is that you want someone who only sees the readme to understand that you did a good, careful job.

Use diagrams

Take-homes offer an opportunity to show how you think and communicate visually.

I used Monodraw to create ascii art diagrams in my code and my readme. I got a lot of comments about it from interviewers.

https://twitter.com/_off_by_one/status/1423342594604601345

Other nice diagramming tools could be Mermaid or Excalidraw (I have feelings about Excalidraw).

Have tests

Write automated tests. I believe that most companies will feel that a lack of tests reflects poorly on a candidate. Focus on tests that are readable and help them understand how you think about the problem space. Great tests read like documentation.

Validate inputs

Good software validates inputs at its perimeter. Sometimes companies will throw test data at your code that they didn't share with you. If there are errors, that's not always a bad thing, but they should be semantic and sensible. You should codify the assumptions you're making not just with tests, but with input validation.

Use modern tech

Don't pull out old libraries and tools. Use the new stuff. That may seem intimidating if you aren't familiar with the latest tooling. Just pick tools that are modern and minimalist, of which there are many. These tools are also easier to read and understand if the interviewer also doesn't know them. It's worth the small time investment to find a cool new library rather than pulling out the same one you used on your last job search.

Negotiation

I'm not an expert in salary negotiation. I'll still share the general framework I used.

"What are your expectations for salary?"

Many companies asked me about my salary expectations. I said something like this:

Look, this is an unusual time. I'm actually not sure what a fair market rate is for me. I've heard of people with similar experience being offered between X and Y^[3] and so I expect I'll get offers in that range.

Some companies just took that information. Others expressed concern because they knew they could not make offers in the high end of my range. In response I told them that I wasn't sure I'd be getting offers that high, and that my range was intentionally large. I also told them that base salary was not the only thing that I cared about and that I would be making a nuanced decision based on many factors.

Lining up offer timelines

At the beginning of the process I picked a date at which point I was hoping to conclude the process. Whenever I had initial conversations with companies I told them this date. Because companies got back to me at different times, sometimes this added a little bit too much pressure, and I ended up withdrawing from the process with some.

After each stage of interview, I tried to refine my understanding of when I might be receiving an offer. Once I knew might be getting an offer from a company, I told other companies about it. I said something like:

Hey first-name!

I wanted to let you know that I'm expecting (🤞) to get an offer from another company in the middle of next week. I've really enjoyed the process with your-company so far, and if it doesn't work out, I would hate for interview logistics to be the reason why. I know I said I was looking to make a final decision by x, but do you think it would be possible to move that up a little bit? Maybe we could conclude the process by y? If this just isn't logistically possible, I totally get it, and I'll do what I can to make things line up. I really appreciate it!

This approach worked every time and was 100% truthful. I would not recommend doing something like this if you're not expecting an offer just to create FOMO.

Comparing offers

This is genuinely hard because predicting the future is hard. If I had gotten red flags from any companies, I would have taken them out of the running, but all the companies I got offers from seemed compelling in one way or another.

I tried to assign a monetary value to everything in a spreadsheet. I quantified uncertainty (bonus, stock performance, etc) as best I could. It was abundantly clear which offer had the highest dollar value. I then asked myself, "do I like any of the other companies enough to pay $x to work there instead of company?" I framed it this way to myself because a dollar not earned is the same as a dollar spent. In my case, it worth paying $x. After telling the other companies about the highest offer, and them not being able to quite meeting it, I took the highest offer I got. That isn't the right choice for everyone in every situation, but it was for me this time.

I'm a few months in, and so far I'm pretty happy! I didn't feel like taking the highest offer was a sacrifice in any way. I wasn't selling my soul. I felt good enough about all the companies and that gave me permission to make it mostly about money.

In summary

This is a great time to switch jobs if you're feeling the itch. Companies are hungry for talent and you can absolutely do really well in a search. But, if your results will vary and I think you'll find that the outcome will mirror the effort you put in. In my case it required some real heavy lifting.

Figure out where your effort can yield the best results and lean into it.

Try to learn about yourself in the process. You may end up with several offers, and there could be subtle cues from your application process that could come in handy. Journal, make spreadsheets, talk to friends.

Communicate openly and respectfully with the companies as the process progresses. Sometimes the truth is the best negotiation technique. Do what you can to make the truth work for you.

When it's all said and done, take care of yourself. This process is exhausting. If you can, take a few weeks off.

My expectations were in part shaped by the fact that my last 2 job searches were short and sweet. ↩︎
In the past when I've reviewed job applications, I've more than once seen cover letters addressed to the wrong company! ↩︎
X and Y were a wide range! ↩︎

Reflections on babies and how they fuck up household supply chains

2022-05-18T00:00:00Z

Having a child is a wonderful, life altering experience. It is profoundly fulfilling and changes you physically, cognitively, and emotionally. Someday maybe I'll have enough distance from it to have something more interesting to say about the emotional part of parenthood, but today is not that day. Today I'm talking about logistics.

What they say is true: having kids is hard. It's hardest on the birthing parent, but it's hard for everyone. The nitty gritty, hands-on-kid childcare happens round-the-clock, and takes a lot of work. But, you're interacting with someone you love.

There's another kind of work that's underestimated by first-time parents (at least by this one): household supply chain. It's just as necessary as any part of childcare, but it's less rewarding than spending time with your new kid.

Stuff needs to get purchased, moved around, cleaned, disposed of, returned, repaired, sold, replaced, etc. This kind of work is always happening in a home, but having a kid absolutely overloads your finite capacity to manage it. You have the number of adult hands you have^[1]. But the throughput of the system goes through the roof. Before you know it, you're living in a pile of dishes, boxes and laundry, asking "what happened" and wondering if FEMA can help.

Accumulation of Objects Under Management

Everything in your home–from furniture to a container of takeout–is an object you're managing. It entered your home because you or someone else thought you needed it. It will leave your home someday. While it's under your care, it's accumulating costs:

Storage cost (unit = m^2)
Cognitive cost (i.e. knowing that it exists, where it is, if it needs maintenance, etc; units = squishy)
Maintenance cost (examples includes washing dishes, doing laundry, having something repaired, etc; units = time, money)

In the normal course of life, objects are entering and leaving your home at a somewhat predictable rate. Sure, you have surges of activity (for example when you move, do spring cleaning, have a big life event, etc). But, your object management is spread out over time. When you have kids you have 10,000 new problems to solve all at once. You need to figure out what to purchase to solve the problem, how to use it, return what didn't work, figure out how to maintain what you keep. No one can tell you a definitive shopping list because every baby, parent, and home is different.

To make matters worse, your kid grows out of most items quickly, so there are aftershocks to the initial supply chain earthquake.

Negative spirals (how assets become liabilities)

There are some negative feedback loops that can happen when you get behind on object maintenance. If everything is clean and put away, you can find it. When you need a burp cloth, or a clean shirt, or both–you know what drawers to open. It's an o(1) operation. When you have piles of clean laundry, you're looking through everything and it's an o(n) operation. In the worst case scenario your dresser itself becomes a derelict object, containing nothing you need and taking up space.

You can get into a downward spiral. The influx of problems causes an influx of new objects. The influx of new objects strains the organizational systems you have in place. When nothing has a home, you're spending more cycles on searching for objects. At the same time, your capacity to work, organize, and maintain is tanking.

Some misc notes/rants on babies and parenthood

Our society is not set up for raising children. It's hard. Even with paid parental leave, even with money, even when you're ready. Childcare is radically expensive. You can't get any fucking baby formula. We're two years into a plague–unprecedented in modern times–and the government is sitting on their hands approving life saving vaccines for kids because.....

It's totally worth it. If you want to have kids, you should go for it because nothing else is quite like it. I just can't write something about babies without saying:

It should be a choice. Abortion should be legal, de-stigmatized, safe and easy to access.
We can afford, as a society, to make it a little easier on babies and new parents.

When you have a kid, this capacity is greatly reduced because one of you is doing childcare at all hours. Also, the birthing parent is recovering for weeks which greatly reduces their capacity. ↩︎

WebReducer Update 3

2022-05-31T00:00:00Z

It's been a little while since my last update on WebReducer. Let's just say I've had a lot going on 😅.

Progress on WebReducer has continued in fits and starts none the less. I've been shockingly productive given how little time I've spent at the keyboard. During parental leave, I've treated this side project like many people would treat reading a book. When I have a few minutes, I pop it open. Reliably I get 30 minutes, twice a week while I'm getting allergy shots. I also find 5m - 1h here and there at all hours of the day and night.

But, there are many unstructured hours every day for my mind to wander. My hands are tied up, but I can think about anything. Sometimes I think about my son, sometimes I think about something that has to get done around the house, and sometimes I think about WebReducer.

This has had a pretty interesting effect on the work. I'm much, much more deliberate. Instead of messing around and trying things, by the time I get to the keyboard, I know exactly what I want to do. I've already considered 3 different approaches, and I've made my preliminary decision. Of course, there are changes I make once I see how things play out, but I generally have a pretty good sense of my system design by the time I try to build it. That's not to say I have a formal plan, in fact, nothing could be farther from the truth, just a slow-cooked mental model of the direction I'm going.

So, what has happened?

First-time UX

When you first open up a WebReducer, you should be able to get up and running fast. Like in a click or two. I built this snazzy little empty-state that lets you configure your first request to the endpoint and copy it to your clipboard in a single click. Here's a janky demo of that working locally.

I like this interface because it gets you into the product without even writing a line of code. I imagine that after you make your first request and see it appear, you'll somehow be prompted to write some code, perhaps by a cursor starting to write some for you... TBD.

Authentication

One of the most important things about WebReducer is that you can use it without an account. When someone goes to webreducer.dev, they already have an endpoint that they can $ curl. This endpoint is ephemeral, and to keep it, you need to sign up.

It is also important that I don't store passwords. I think passwords are a barrier to entry and a security risk. I want magic link authentication. Eventually there can be a 2FA option (which could be a password, authentication app, etc). For the MVP I don't think this is necessary.

There are 2 parts of WebReducer authentication: sign-in tokens and credentials.

Credentials are given to all users–authenticated or otherwise. The second you hit the app, you have a guest account that's not associated with any email address. Once you sign up, we send over a sign-in token to your email address. When you go to the URL we send, we consider to be a "real" user. Sign-in tokens exist to prove you control an email address.

Credentials are made up of a short-lasting JSON web token and a long lasting refresh token. Both are sent along with every request you make. JWTs can be validated statelessly by multiple independent micro-services, and since the app is relatively chatty this has certain advantages. We don't need to do a db hit on every request or do any kind of shared session state. But, when the JWT expires, the refresh token is used to issue a new JWT/refresh token pair. The refresh token is stateful–it exists in the database. Refresh tokens can only be used once.

To sum up:

You can use the app without signing up
Once you sign up, you never need a password
You don't get logged out all the time
I can spin up microservices that can use this auth scheme without holding centralized state

Secrets 🤫

I realized that many, many webhook endpoints don't work without stored secrets. Often, webhook providers want to you validate the authenticity of a message using hash-based message authentication code (HMAC). HMAC requires you to have a secret in your possession that was given to you by the provider. It'd be terrible for me to require users to just keep that secret in plain text in their code. I don't plan on having my database breached, but if I did, I'd want it to be harder than that to access this kind of secret.

I decided to build a separate microservice in charge of secrets.

flowchart LR
    Client[Remix App]
    Browser
    HookClient[Endpoint Clients]
    subgraph privateNetwork[Private Network]

        Server
        Redis[(Redis)]
        Postgres[(Postgres)]
        SecretsDb[(SecretsDb)]
        Worker
        Secrets[Secrets Service]
        subgraph isolated-docker
            Runner
        end
    end

    click Client "https://github.com/zekenie/web-reducer/tree/main/client#readme"
    click Server "https://github.com/zekenie/web-reducer/tree/main/server#readme"
    click Worker "https://github.com/zekenie/web-reducer/tree/main/server#readme"
    click Secrets "https://github.com/zekenie/web-reducer/tree/main/secrets#readme"
    click Runner "https://github.com/zekenie/web-reducer/tree/main/runner#readme"


    Client <--> Server
    Server --> Secrets
    HookClient <--> Client
    Secrets <--> SecretsDb
    Server <--> Redis
    Server <--> Postgres
    Worker <--> Redis
    Worker <--> Postgres
    Worker <--> Runner
    Browser <--> Client
    Redis

classDef selected stroke-width:4px;
class Secrets selected;

The organizing principal behind this decision is that if any single part of the system is breached, secrets are not compromised.

Secrets are organized in namespaces. A namespace is simply a collection of secrets with an access key. There is no way to list access keys, the Secret Service doesn't even have them, it just stores a hash digest. When a WebReducer is made, there is a namespace created, and the access key is stored encrypted in the monolith. If the monolith db is breached, an attacker would also need an encryption secret which is stored in an environment variable.

If the secret database itself was breached, it would be hard for an attacker to exploit. Every key value pair is encrypted with a per-namespace key, and these keys are encrypted with a key that's stored in an environment variable.

I'm working on automatic key rotation for all these systems, but it's not there yet.

I'm sure there are flaws in this system. I'm not trying to launch this product with enterprise-grade security. I'm trying to not be irresponsible as I learn if this product has legs. I think if people start using it, it'll get better. It's open source. There will eventually be a bug bounty program. If this product is meant to be, security will come. Customers will be warned.

New ideas

Here's an incomplete list of feature ideas. Some of these I may get to, some I may not. Some are distractions.

Side effects. WebReducer code is synchronous and functionally pure. With the same input, and the same code, you get the same output. There is currently no communication with any outside system. A side effects feature would allow users to trigger requests to other systems after the reducer runs. Unlike the reducer, side effect code would not re-run with code changes. Side effects could be an email you conditionally send to yourself when the state changes in a certain way, an http request you send to a slack endpoint to notify a channel, etc.
State branches. You should be able to have a single WebReducer control different versions of your state. You could map different write keys to different state branches
Scheduled requests. What if you could set up a cron to hit your own WebReducer. Could be very useful for some applications
Queryable views. The original WebReducer concept had readKeys and writeKeys These were effectively revokable endpoints that could read and write state. writeKeys couldn't mutate state directly, instead state changes were mediated by the reducer that users wrote. I realized that it could be great to have the read behavior also be configurable. Perhaps there's a readState function that is called on every readKey request.

Demo ideas

I really need to focus and get this product into the world. I'm prone to distractions. One source of focus for me has been thinking about demos. When I consider what I will build with this product, it helps me understand what features are essential and what's cruft. Here are a list of demos I'm itching to make. I think I'll live code them once I get a version of this product out there.

Card games. I could build realtime, multiuser card games like blackjack.
Simple form submission. One of the coolest applications of WebReducer is a form submission. You can point an HTML form at your endpoint and submit a form. With a responder function, you can 301 redirect users back to your site or render an HTML thank you page. I love this demo because it's really really easy and practical.
Interactive blog post. In a blog post like this, there could be little buttons next to ideas like (neat!) that users could click to tell me they dig something
Personal dashboard. This was kind of the idea that got WebReducer going in the first place. I wanted a public URL in my email signature that would answer the question "how's it going..." I wanted to show my internet speed, sleep quality, etc. Recently I thought it'd be cool to show % of uncalendared time, # of unread emails, etc. This is a very webhook-heavy application, it doesn't require a relational database, and is perfect for WebReducer.
Dynamic consultant pricing. I'm not a consultant, so that makes this somewhat theoretical. But, if you wanted to book an hour of my time, perhaps there could be an algo aware of my calendar, inbox status, mood, etc.
Bloom filter. I realized that I could write a hacky little bloom filter like this. This is obviously a weird use case, but it's just kind of cool that it's possible. There are probably cousins of this idea that are cooler.

Any ideas for me? Want to be a beta tester? Give me your email

Elevators, Subways, and Rocket Launchers: The Unpredictable Path of Software Development

2023-09-04T00:00:00Z

There's been plenty of ink spilled on the topic of what to call software engineers. Hackers? Coders? Developers? This semantic debate has been mostly boring to me until recently. It's hard to find one term to use to describe our job because our job can change so much from day to day.

When you're building at scale, the job can superficially resemble classical engineering. You need to build a model of the system based on empirical data, understand the constraints of the system, and build with enough flexibility to avoid a range of disasters.

On the other hand, if you're building a new product that has no users, you use a profoundly different toolkit. The product itself is a hypothesis. You're trying to make something useful that doesn't yet exist. There's no reason to believe the thing your building will solve a real problem. You need to figure out how to build something as quickly as possible without contaminating the experiment. If you make it too cheaply the jankyness will get in the way and no one will use it, even if the idea is great. But, if you decide to use Kubernetes and micro-services with mature observability, you'll never stand anything up at all.

I've worked at startups and now I've worked at a couple bigger companies for a few years, too. I've been surprised to learn that you still need "hacking" at large organizations. Some projects–a prototype of an internal tool, for example–call for "hacking" while others that will be trafficked by millions of people need "engineering."

This one dimensional spectrum that I've described with "hacking" on one side and "engineering" on the other is still too simple.

Some days I'm a detective or a spy, trying to understand how someone exploited (or could exploit) a system. How do we know someone is who they say they are? Could anyone sneak past our security? How would we know that they did? What's the worst thing that might happen if they fool us?

Other times I'm wearing the hat of an ER doctor trying to understand the pathophysiology of the system in distress. "What could cause X reading to be elevated with Y symptom.... who should we page in for a consult? What can we do to stabilize the patient while we work on a diagnosis?"

I've occasionally worked on "core" projects where my goal is to support other engineers in what they build. When I'm doing this I need to consider the emergent properties of a team of hundreds of people, slowly turning over over decades. People don't have perfect visibility or discipline. How can we create incentives for people to do the "right" thing? How can we make it as easy as possible to modernize legacy surfaces. On those days it feels like I'm almost an urban planner, economist or even a politician.

As software engineers, the things we build are unlike anything else out there. They're in constant flux and can change drastically over long periods of time. Imagine if the people building an elevator system in a skyscraper found out a decade later that they were actually building a subway system! That's what it sometimes feels like. The decisions that were made when you were making the elevator don't always hold up in hindsight. The context has shifted 90 degrees.

And yet, you can't start over–a million people are using your subway every day. You just have to move forward with an eye toward the future. For all you know you might need to make this thing into a rocket launcher a decade from now.

If I can leave you with one piece of advice... No one tells you when it's time to switch toolkits, and and it's necessary. If you miss the moment when it's time to change things up, your project will be in trouble. This job requires enormous flexibility. Ask yourself and your team, "what tools do we need for this project?" or "is it time to switch hats now that this thing is out in prod?" There's a lot to learn from different disciplines and from each other and we're still at the beginning.

Nextty: a radically collaborative computing environment

2023-11-27T00:00:00Z

I want to build a community computing platform. It's remote-friendly, but synchronous. It's orders of magnitude more immersive and cooperative than screen sharing. You could call it a chat app where you also write code... or a futuristic repl with your buddies. It's a computational space where you solve problems together. As people start working in this kind of environment, the lines between programming and other communication blur. Everyone in the community ends up writing, or at least invoking, little programs all the time, even if they're not "software engineers."

There's a transition that's just begun in computing that makes this idea suddenly and increasingly viable. The cost of expressing computational thought has begun an exponential decline (just like solar energy, light, and transistors) due to recent innovations in generative AI. The population of people capable of performing computational tasks is going to explode, but not in today's computing environments.

One of the defining characteristics of communities is that they inhabit spaces. Communities can be made up of one or more spaces that are visually and functionally distinct and evolve over time. User interface elements can be built to represent common computational objects. Objects can be manipulated in multiple ways. Someone with more of a computing background might choose to manipulate an object in a lower-level way. But other users might manipulate the same object through a more traditional user interface.

No one's ever said "I love what you've done with the place" about a slack channel or a zoom call. If you've ever worked in a science lab with a healthy culture, you know that people do say that about each others lab benches. People pin things on the walls that reflect their personality. People draw doodles on each others' whiteboards while they're out at lunch. People are social. People personalize the form and function of spaces if you let them. Remember MySpace?

So what would a specific room, or channel look like? Well it would be a folder on a computer somewhere.

./example-channel
├── access.toml
├── agents
│   ├── decision-summarizer.md
│   └── visualization-helper.md
├── channel.sqlite
├── channel.toml
├── community.toml
├── profile.ts
├── readme.md
└── views
    ├── cover.html
    └── task-bar.html

access.toml would specify who can read/write/evaluate/search/mention/react/etc
agents would be a directory for LLM-backed bots. There would be some configuration specified in front-matter. You could also write non-LLM agents too. Communities can install plugins that make certain LLM providers or open source libraries available to every channel in the community so you don't need to set it up
profile.ts is a set of code that runs every time the channel node boots up. It's for configuring common commands (think .bashprofile)
readme.md opens in the sidebar of the UI by default. It's a cover page to orient onlookers
channel.toml specifies some basic configuration and rules
community.toml is autogenerated and has details about how the channel connects to the community server.
The views directory contains html files (I think?). You can overwrite system partials and customize UI here. Very much TBD.

Not every channel would need to be on the same hardware. Some communities would run all on one box, but in other cases, you might want to have a channel running on your own laptop. That way, when you're working in "your office" it's really fast. Plus, you can use it on an airplane. The software itself isn't prescriptive about how this is done.

If you're having trouble imagining this applying to a chat-repl, that's OK. It's hard to imagine. It's different from any computing environment you've ever been in. It's social and vibrant. It's a place to gather, learn and teach. It's a breeze to pick up but extraordinarily deep and rich. You can get lost in it.

OK, let's try a thought exercise. Out of all the digital communities you've been a part of, what was the best one? Now imagine what that community would be capable of if instead of passing around links to various vendor-backed web apps on slack, people could exchange references to computational objects in the digital co-working space. There would be common abstractions for querying and visualizing data across all kinds of objects. There would also be batteries-included tools for what we'd conventionally call "telemetry." The entire computing environment would be highly observable and visible. The entire organization would share a common set of tools for seeing change over time.

Everything would default to open. If you're working in a science lab, you can see what's on your coworkers desk. In this community, you can also see what's on your coworkers desk. In certain situations, that kind of openness doesn't make sense. Not every room has the same permission structure. There are locks and keys. Some people are read-only in some spaces. Some people can execute commands. Some people don't even know the room exists. But that's not the norm. The norm is open.

For some communities, security and access control would be quite important. And while it's true there's a security burden on the community, today's conventionally run organizations have enormous security liabilities as well, they're just just somewhat obscured by vendors. Instead of your data being in one place where you can control it and monitor it, it's in who-knows how many organizations hands.

While this type of computing is decidedly expensive, the software behind it would have to be free and open source. The community software platform offers tremendous value to organizations, and surely that value can be captured by people offering to help build these organizations. But, in order to make computational spaces available to all kinds of people and organizations, the foundational software would have to be free like water. It would have to be something that you could set up on a VM in 5 minutes and get started with. But it would have to have enough complexity to allow communities to grow and evolve over time into elaborate spaces that suit various needs.

Note: this isn't a hyperscale idea. We don't need the software to scale gazillions of users because no community will be that big. Because this software is run by the organization, even if they pay someone to help, the software platform only needs to accommodate the largest types of communities. This smaller scale has implications about the platform. More will be done with flat files. If there's a need for a database, SQLite will get used and it's baked into the platform.

Other note: if you're thinking "well this would never work at ___" you might be right. I can't imagine the security team at my day job ever saying yes to this. But not all computational communities look like your workplace. And I think over time as the software matures, it might support more and more types of organization and you might find yourself surprised who starts using it.

So what's next? well... I think I need to start by getting an MVP together. Once I get some of the essential pieces in place, I hope to start developing on the platform. If the code is on github, it'll be a side effect. To get a sense for how this feels, I'm already using a chat app to jot down ideas and to see how things feel (it feels good!). If you want to get involved give me a shout out on the dead bird app. I'm gonna migrate somewhere else soon, but haven't yet.

Book: Mindstorms: Children, Computers, And Powerful Ideas by Seymour Papert

Have a ball of mud? You need(ed) a map.

2024-02-18T00:00:00Z

At work right now we’re currently working on modularizing our ~15 year old rails codebase. One conversation topic comes up frequently: is there ideal size range for a module? Should we make many small modules that are precise, or fewer course grained modules (or a mixture)? Should we have multiple tiers of nested modules?

Maybe we should take some design inspiration from “the city”.

Cities are big and complex! Despite this, people seem to get by pretty well in cities. We have tools to manage the level of complexity in our daily lives. Cities are continuously being constructed, just like software, by a team that never stops rolling over. The people who started it aren’t around anymore to say why every little thing is just the way it is. Cities are flawed and cities are beautiful.

How do people wrangle the complexity of a city at a practical level? For one, we have maps. Millions of tourists visit New York City every year from all over the world. By and large, they manage to use the subway because of its incredible map. The map over-simplifies here and there, and makes some compromises. But, at its core, it achieves its purpose. It’s navigable by a tourist, with some effort.

If the people building your software are struggling to navigate when they onboard, or perform day to day tasks, chances are you need a map. There’s a bit of a catch-22, as well: if you’ve been building software without a map for 15 years, you haven’t had the tools to design well, so your first map might look like a ball of mud.

This brings us back to the question at the top. When we’re refactoring a ball of mud, how big should we be chunking? Let’s look to the city.

A single module should have the same level of complexity as a home or even apartment building would. It shouldn’t need a map to understand. Maybe there’s a blueprint from when it was built? But not a map for every day use. Even if a module has some design warts and compromises, you shouldn’t need a map to explain it.

A set of modules representing an entire domain are sort of like a neighborhood. A neighborhood could have a map, especially for people who aren’t very familiar with the geography. But it shouldn’t be an essential tool for people who live and work there. They should be able to go off of memory for the most part.

All big cities have maps. They wouldn’t function without them. Can you imagine what NYC would look like if millions of tourists descended without maps?

Codebases that don’t have maps over time tend to function like cities without maps. No one can agree were the borders of anything are. You can’t communicate your future plans relative to anything so people’s intentions are always colliding. If you try to turn to graph force diagrams, it just looks like a big ball of mud. I think that’s what we call a chronic condition… It’s manageable, it can improve, but it’s not really ever going to go away as a problem.

Why can’t the map be the territory? What if our programming medium was inherently mappable. I’m not saying we’d write code in a visual medium, per se, or that we’ll be opening up our IDEs in VR. But I’m proposing that instead of first using the file system as a way to navigate code, we used some kind of purpose-built map.

If the mapping tools were good enough, that engineers started to rely on it to make better decisions.

Last week at work I was asked to spec out a greenfield project. I used D2Lang, as I’ve been known to do of late. I can’t show you the real diagram but it only looked slightly more complicated than this

In d2, you can produce this image with the following code:

http: {
  users: {
    sign_up
  }

  tasks: {
    create
    assign
    complete
    defer
  }
}

# creates a user
http.users.sign_up -> users

# ...
http.tasks.* -> tasks

tasks: {
  shape: sql_table
  id: int {constraint: primary_key}
  last_updated: timestamp with time zone
  assignee_id: int {constraint: foreign_key}
  state: string
}

users: {
  shape: sql_table
  id: int {constraint: primary_key}
  name: timestamp with time zone
  email: int {constraint: foreign_key}
}

tasks -> users

Then an idea hit me. What if I had a program in my $PATH that could run this as code.

$ magic-cli ./prototype.d2 --output deno

It could clearly build a sqlite database from those sql tables. With some descriptive comments a garden variety gen-ai model could write some prototype code (and even some tests) for individual nodes.

When you want to make big architectural changes, you don’t try to mutate the existing code into the new shape, you instead start over at the map to make a design decision about the intended change. Sure, a bunch of glue code will get wiped away with , and the new code will need to be extensively tested. But, because the map has been automatically updated with every manual change, you’re not starting frame square 1.

Essentially, you’d “GPU compile” down this D2 + english comments + prompt into a nice little Deno binary with a little human in the loop to make sure it does what it’s supposed to.

I am actually interested in hacking on this, but I think its a long way off from anything you can use. But, if you’re interested in working on it with me, let me know!

We need to be the historians of our own software

2024-03-09T00:00:00Z

Codebases are peculiar things. They’re accumulators of debt and investments, warts and jewels. Over time, all the decisions–conscious and unconscious–made by a rotating cast of characters sum up to the current state of your main branch, HEAD. The HEAD may change every few minutes, hours or days. But as long as your codebase is alive, it’s changing.

While every codebase is in motion, some things are immutable (or change so slowly they feel that way). Software can have character and a sense of place, much like a city. And just like a city, the bones of a codebase are built with imperfect context and are adapted over generations, with mixed results.

When I close my eyes, the codebase I'm working on almost has a smell to it. If I were to go away away for a few years and come back, this codebase would be different, but still itself.

nyhistory.org

When you make a change to your codebase, you’re not starting from zero. Sometimes it can feel like you’re standing on the shoulders of giants. Other times it feels like those who came before you made foolish choices. Either way, they defined the semantic water you’re swimming in, and they’re more like you than you might think: human beings trying to make something makes sense with imperfect tools and imperfect craft.

If we want to influence the future of our software–to make it a better place to live and work in–we need to understand where it came from. We need to understand the human story of how we got from the first commit to HEAD. We need history.

When it comes to historical tools, we have a lot, actually. Git in particular comes to mind as an asset. Git is like an archival record of every little change over time. It can answer specific questions and contains many treasures if you take the time to look. For example, if you’re wondering how a file came to be, git has a lot to say, especially if there are commonly good PR descriptions in your project.

But it’s really hard to contextualize a long series of git commits. You can’t zoom out and understand the the world around the code. In 2012, what did the app feel like from the point of view of a user? What did the codebase feel like from the point of view of the engineers? Try checking out a historical commit and running it–If it’s even possible, it’s not fun or practical. Beyond the software itself, the human stories are totally out of reach. What was on the minds of the engineers? What were their struggles? Their triumphs?

Depending on the age of your project, you may be able to speak with some people you see in the git blame. You should. You should ask them questions and write down what they remember. It’s gold. But, that’s not enough. Gray matter is an imperfect storage medium. Memories fade. You’re not talking to the version of your colleague who just made a decisions, you’re talking to the person who has the benefit of hindsight. It’s not possible for them to reconstruct their cognitive state. Even if it were, you’d just be talking to one person. Most of the team has turned over.

We would benefit from tools we don’t have

If we really want to understand the codebases we work in, we need better tools to construct the history. We need software to sift through git. I want to know how the quantities of certain types of code changed over time. I want to know when one style came into the codebase, plateaued and then declined. I want to understand trends.

But beyond git, we need to understand the people. How did teams and team members change over time? What were they trying to do, in their own words? where did they think they were going? What was contentious? What was going on in the business? What was going on in the world?

There are many other artifacts that are not in git. Businesses always have a large set of documents in some kind of wiki or docs editor. They usually have author attribution and a big set of hyperlinks to other artifacts. This web is dense. The authors are very present-minded. They don’t include a lot for future readers.

In the last 18 months, it’s gotten a lot easier to automatically summarize documents. We should be using Generative AI to make these artifacts easier to use.

I want a timeline view of a project’s history. I want to be able to pan and zoom. I want to go to different points in time and see product demos from the teams that worked on it. I want to have links to various documents with well picked excerpts. I want summaries. I want to understand quarter-by-quarter what the people were trying to achieve. I want to see when people joined and when they left.

Ultimately making history is an act of synthesis. What I’m picturing in my head isn’t the history itself, it’s a tool to make the history easier to distill. It would make a great deal of data about the past something that the whole team can understand and benefit from.

History can provoke hard, necessary conversations

Just like every other historical endeavor, constructing a narrative of the past can provoke difficult conversations. Why did we suddenly change all our engineering practices in 2015? Oh it’s because the whole team quit when the VPE was fired. Woah. Ouch. Why did we go all-in on technology? Oh, it’s because person (who may still work here or know people who does!) thought it was a great idea.

In order to make this endeavor useful you need to adopt a nonjudgmental mindset. You should assume that the people who built this codebase were making the best decisions they could with the tools and information they had at the time. You may feel like better decisions are obvious, but you’re benefiting from information they didn’t have.

And, sometimes these difficult conversations are necessary. Sometimes in order to move forward we have to be open about what really happened, and own it. History isn’t always rosy, and that’s OK.

People already have narratives of the past, they're just not always backed by data

Humans make narratives like spiders make webs. Everyone in your organization has a story they tell themselves about the past. Someone might think "oh quality really dipped as we focused on growth pre-IPO" or "we really sped up when we got our new CTO" for example. But did these things really happen? These narratives reflect an individuals lived experience, and we shouldn't discount that. But can we observe anything empirical?

It's absolutely possible to over-index on empirical data... and it's important to be mindful of that. But empirical data, when it is high quality, can help us zoom out and show a larger perspective than one individual on one team. It can mitigate fuzzy memories and group-think. It can show how parts of a system interacted over time. It can help us learn from the past, correct misconceptions, and make us overall better at whatever it is we're trying to do.

You can make it easier for future historians (yourself?)

Tomorrow's history is now. You an do things now that will make life easier for future historians. At work we have a repo called artifacts which contains all sorts of runtime artifacts that would be difficult for us to reconstruct later without the app running. As an example, we have an artifact that gives us details about our ActiveRecord models. We know for each commit how many public instance methods existed any given model, or what AR associations and scopes were defined. We also know what Packwerk pack each model is associated with, and which team is the owner. For our Packwerk modules, we know each of their "modularity scores"–an in-house metric.

These artifacts are tracked in git and are updated in a "shadow repo" every time our monolith builds. This means we can see how they change over time. We've had some of these artifacts for 8 months, and they've proven invaluable.

But we could be doing more. We could be conducting interviews. We could be journaling. We could be proactively making the lives of future historians easier, and we should.

Get shit done by warping friction-space

2024-05-29T00:00:00Z

I have a process for making a codebase better over time, and it's so simple it could almost be a tweet:

Make stuff that should be easy easy. Make stuff that should be hard hard.

If my brain had telemetry, I think the CPU load while at work would look something like this. Noisy and spiky.

The spikes are part of what people call "friction" at work. Friction isn't always bad, but usually it is. There are a handful of things that you actually want to require extra effort to do. But as a rule it's something we are constantly trying to minimize.

My process starts by doing product engineering. Every time I pick something up, be it exciting or mundane, I take telemetry. When something seems really hard, I bookmark it.

Periodically, I reflect on what's been hard. Here are some made up examples

%%{init: {"themeVariables": {"quadrant1TextFill": "darkgreen", "quadrant3TextFill": "darkyellow", "quadrant4TextFill": "orange", "quadrant2TextFill": "red" } }}%%
quadrantChart
    title Friction Evaluation Matrix
    x-axis Was Easy --> Was Hard
    y-axis Should Have Been Easy --> Should Have Been Hard
    quadrant-1 Awesome work
    quadrant-2 Reckless zone
    quadrant-3 Necessary toil
    quadrant-4 Frustrating bullshit

    Necessary refactoring: [0.45, 0.17]
    Writing tests: [0.1, 0.13]
    Adding a module: [0.7, 0.23]
    Adding a microservice: [0.88, 0.33]

    Architecture project: [0.87, 0.79]
    Building consensus: [0.7, 0.59]
    Adopting new technology: [0.6, 0.69]
    Mentorship: [0.62, 0.8]


    Add another ORM Relation: [0.20, 0.64]
    SQL against prod: [0.15, 0.72]

Both axes are subjective

But, the X-Axis is a qualitative measurement of your experience–what was and was not hard. The Y-Axis is more subjective. It's your opinion about what should be.

Later, when I'm doing self-driven work, I try to react to my observations.

Address The Reckless Zone

If I observe something in The Reckless Zone, it's important to address quickly. For example, if engineers at a small but growing startup are running queries against the main production DB, that's something that needs to be addressed because it could take down the system.

To mitigate observations in The Reckless Zone, we need to add friction. It's important that the friction is automated. I'm talking about linters, PR bots, Slack alerts, etc. We should minimize the extent to which someone needs to enforce stuff.

Celebrate awesome work

It's important for ICs' work to be visible and celebrated. When folks do good, hard things, the broader team needs to know. Different companies have different mechanisms of this, but usually there's at least a slack channel for celebrating wins. If not, maybe there should be.

Work on Frustrating Bullshit

Whenever you encounter something that is hard but should be easy, it's important to take a careful look at it. This could be a golden opportunity to improve the broader tech org.

Not everything is easily addressable. Some problems are not technical ones. But this is the quadrant that's holding you back. You need to build a culture that chips away at this unnecessary friction over time, and periodically makes investments that produce big leaps forward.

When teams are "too busy" to make investments in working better, that means they'll be just as "busy" next year. They probably feel chronically overworked and frustrated. Imagining shoveling a huge pile of dirt with a spoon and thinking "there's gotta be a better way," only to be told "nope, don't think about that! too much dirt!"

The org is a dynamic system

Different people in the organization will have different perspectives on what needs to be hard and what doesn't. It's a big dynamic system. It's hard work to build enough consensus, but it's worth it. When you achieve a well-calibrated friction space, goals are more likely to get met, people are happier, and the company starts to trust that the engineering organization can do what it says.

Let's take a minute to zoom in on a subgraph here...

graph BT
    credibility["Credibility"] -->|++| tech_investments["Tech Investments"]
    linkStyle 0 stroke:green,stroke-width:3px
    tech_investments -.->|----| friction[Friction]
    linkStyle 1 stroke:#f22,stroke-width:7px
    tech_investments -.->|++| friction
    linkStyle 2 stroke:green,stroke-width:3px
    tech_investments -->|++| reliability["Reliability"]
    linkStyle 3 stroke:green,stroke-width:4px
    tech_investments -->|++| security["Security"]
    linkStyle 4 stroke:green,stroke-width:2px
    tech_investments -->|--| capacity["Capacity (gas)"]
    linkStyle 5 stroke:red,stroke-width:2px
    tech_investments -->|+++| cohesion["Conative Cohesion"]
    linkStyle 6 stroke:green,stroke-width:4px
    tech_investments -->|-| surface_area["Surface Area"]
    linkStyle 7 stroke:red,stroke-width:1px
    tech_investments -->|++| mpg["Milage (mpg)"]
    linkStyle 8 stroke:green,stroke-width:2px
    tech_investments -->|++| documentation["Documentation"]
    linkStyle 9 stroke:green,stroke-width:2px

    capacity -->|++| tech_investments
    linkStyle 10 stroke:green,stroke-width:2px

    documentation -->|+| mpg
    linkStyle 11 stroke:green,stroke-width:2px

    reliability -->|++| credibility
    linkStyle 12 stroke:green,stroke-width:2px

    mpg -->|++| credibility
    linkStyle 13 stroke:green,stroke-width:2px

    security -->|++| credibility
    linkStyle 14 stroke:green,stroke-width:2px

    friction -->|----| mpg
    linkStyle 15 stroke:red,stroke-width:5px

Tech Investments

Tech investments take away capacity you could spend elsewhere. These investments add and remove friction, but they take away more than they add. They enhance security, reliability, and conative cohesion. Tech investments (in contrast to product investments), tend to modestly remove surface area (a liability). Tech investments empty your gas tank, but increase your milage.

In order to do tech investments, you need some credibility in the org, but by doing tech investments well, you tend to build credibility, as well.

Cognitive cohesion

Conative cohesion is a mostly squishy idea that things that belong together are found together. At work, we try to measure this by seeing how many modules need to be modified in a single chunk of work. If for most tasks you can open up a single module, load it into your brain, and ignore the rest of the application, that's a proxy for high cognitive cohesion.

Another way to get a feel for cognitive cohesion: how many people have to be involved to make a seemingly simple change happen? Is it a couple people on a team, or 4 teams coordinating via weekly "scrum of scrum" meetings? At prior company, creating a new promotional discount took a quarter and required constant syncing of more teams than I'd care to admit. That was because the details of discounts were spread out into a web of tiny, brittle micro-services that had low cognitive cohesion.

graph BT
    tech_investments[Tech Investments] -->|+++| cohesion["Conative Cohesion"]
    linkStyle 0 stroke:green, stroke-width:4px
    cohesion -->|--| friction
    linkStyle 1 stroke:red, stroke-width:3px
    cohesion -->|+| morale
    linkStyle 2 stroke:green, stroke-width:1px
    cohesion -->|++++| code_reads["Code Read MPG"]
    linkStyle 3 stroke:green, stroke-width:5px
    cohesion -->|++| code_writes["Code Writes MPG"]
    linkStyle 4 stroke:green, stroke-width:2px
    documentation -->|+| cohesion
    linkStyle 5 stroke:green, stroke-width:1px

Velocity

It's tempting to think of product investment and tech investment as zero-sum. An hour spent on one is an hour not spent on the other. But friction (in the wrong places) has a profound drag on product velocity.

On the other side of the coin, it's not really possible for engineers to achieve business outcomes with tech investments alone.

graph BT
    friction -->|---| velocity["Product Investment (m/s^2)"]
    linkStyle 0 stroke:#f22, stroke-width:3px
    velocity -->|+++| outcomes
    linkStyle 1 stroke:green, stroke-width:5px
    capacity -->|++| velocity
    linkStyle 2 stroke:green, stroke-width:3px
    reliability -->|+| velocity
    linkStyle 3 stroke:green, stroke-width:1px

Changing friction space

The real task of senior-level ICs is to continuously (and strategically) mold the friction space in response to the needs of the business. The reason why this is continuous is because the needs of the business are in constant flux. A well-calibrated friction space for a scrappy 10 person engineering organization will be totally dysfunctional for a 20 or 200 person org.

I'm not a big fan of dedicated teams that are charged with reducing friction. It can be the right move, but these teams tend to lose touch with the reality of the friction space and work off of their mental model of the friction space which drifts away from reality. Instead I'm in favor of empowering every engineer to work on the friction space. If there is a dedicated team, it's important for them to have good feedback mechanisms. This could take the shape of pair programming with "product engineering" or rotating onto teams for a quarter per year.

If you're company hasn't learned how to manage it's friction space yet, work on it. Things will only get more painful until you figure out how to manage friction as part of development.

Modeling a big spaghetti app

2024-07-07T00:00:00Z

The problem

At work, we've been modularizing our old Rails monolith for the better part of 2 years. We've made a ton of progress and I've personally learned a lot. Unfortunately, it's been challenging to share details publicly because all the examples are closed source, and sensitive. If I wanted to write any open-source any visualization tooling, it'd be hard to test on anything resembling a real project.

I recently made some progress in this problem space. LLMs are great at synthetic data generation, and it turns out that Claude 3.5 Sonnet can, given the right combination of prompts, produce a pretty mean looking spaghetti ball monolith.

First, I asked Claude to generate me a ball of mud model layer.

I didn't feel like this was "monolith-y enough" so I coaxed it to get more tangled. I also asked for it to provide edge descriptions and a few different user types.

That's what I'm talking about.

Then, in a new session, I asked if Claude could generate a graphql schema for this app. It did a good job on it's first attempt. I asked it in a subsequent request to add documentation to the schema. I also needed to tell it to use relay... I ended up with this:

# Root Query type
# Provides entry points for reading data from the system
type Query {
  # Retrieves a user by their unique identifier
  user(id: ID!): User
  # Fetches a product by its unique identifier
  product(id: ID!): Product
  # Retrieves an order by its unique identifier
  order(id: ID!): Order
  # Fetches a category by its unique identifier
  category(id: ID!): Category
  # Retrieves a support ticket by its unique identifier
  supportTicket(id: ID!): SupportTicket
  # Fetches a warehouse by its unique identifier
  warehouse(id: ID!): Warehouse
  # Retrieves a paginated list of products with optional filtering
  products(first: Int, after: String, filter: ProductFilterInput): ProductConnection!
  # Fetches a paginated list of orders with optional filtering
  orders(first: Int, after: String, filter: OrderFilterInput): OrderConnection!
}

# Root Mutation type
# Provides entry points for modifying data in the system
type Mutation {
  # Creates a new order
  createOrder(input: CreateOrderInput!): Order
  # Updates an existing product's information
  updateProduct(id: ID!, input: UpdateProductInput!): Product
  # Creates a new support ticket
  createSupportTicket(input: CreateSupportTicketInput!): SupportTicket
  # Updates the inventory quantity for a product
  updateInventory(id: ID!, quantity: Int!): Inventory
}

# Represents a user in the system
type User {
  id: ID!
  email: String!
  profile: Profile
  # Paginated list of user's addresses
  addresses(first: Int, after: String): AddressConnection!
  # Paginated list of user's orders
  orders(first: Int, after: String): OrderConnection!
  # Paginated list of user's reviews
  reviews(first: Int, after: String): ReviewConnection!
  wishlist: Wishlist
  shoppingCart: ShoppingCart
  # Paginated list of user's notifications
  notifications(first: Int, after: String): NotificationConnection!
  # Paginated list of user's payment methods
  paymentMethods(first: Int, after: String): PaymentMethodConnection!
  # Paginated list of user's membership plans
  membershipPlans(first: Int, after: String): MembershipPlanConnection!
  role: Role
  loyaltyPoints: Int
}

# Represents a product in the system
type Product {
  id: ID!
  name: String!
  description: String
  price: Float!
  category: Category
  inventory: Inventory
  # Paginated list of product reviews
  reviews(first: Int, after: String): ReviewConnection!
  # Paginated list of product images
  images(first: Int, after: String): ImageConnection!
  # Paginated list of product attributes
  attributes(first: Int, after: String): AttributeConnection!
  # Paginated list of related products
  relatedProducts(first: Int, after: String): ProductConnection!
  supplier: Supplier
}

# Represents an order in the system
type Order {
  id: ID!
  user: User!
  # Paginated list of order items
  orderItems(first: Int, after: String): OrderItemConnection!
  totalAmount: Float!
  status: OrderStatus!
  shippingAddress: Address!
  billingAddress: Address!
  # Paginated list of payment transactions for this order
  paymentTransactions(first: Int, after: String): PaymentTransactionConnection!
  # Paginated list of shipments for this order
  shipments(first: Int, after: String): ShipmentConnection!
  createdAt: DateTime!
}

# Represents a product category
type Category {
  id: ID!
  name: String!
  parentCategory: Category
  # Paginated list of subcategories
  subcategories(first: Int, after: String): CategoryConnection!
  # Paginated list of products in this category
  products(first: Int, after: String): ProductConnection!
}

# Represents the inventory of a product in a specific warehouse
type Inventory {
  id: ID!
  product: Product!
  quantity: Int!
  warehouse: Warehouse!
  # Paginated list of stock movements for this inventory
  stockMovements(first: Int, after: String): StockMovementConnection!
}

# Represents a warehouse in the system
type Warehouse {
  id: ID!
  name: String!
  address: Address!
  # Paginated list of inventories in this warehouse
  inventories(first: Int, after: String): InventoryConnection!
  manager: User
}

# Represents a customer support ticket
type SupportTicket {
  id: ID!
  customer: User!
  subject: String!
  description: String!
  status: TicketStatus!
  agent: User
  # Paginated list of responses to this ticket
  responses(first: Int, after: String): TicketResponseConnection!
  createdAt: DateTime!
  updatedAt: DateTime!
}

# Represents a product supplier
type Supplier {
  id: ID!
  name: String!
  # Paginated list of products supplied by this supplier
  products(first: Int, after: String): ProductConnection!
  qualityRating: Float
}

# Represents a product review
type Review {
  id: ID!
  product: Product!
  user: User!
  rating: Int!
  comment: String
  createdAt: DateTime!
}

# Represents a payment transaction
type PaymentTransaction {
  id: ID!
  order: Order!
  amount: Float!
  status: PaymentStatus!
  paymentMethod: PaymentMethod!
  createdAt: DateTime!
}

# Represents a shipment for an order
type Shipment {
  id: ID!
  order: Order!
  trackingNumber: String
  status: ShipmentStatus!
  estimatedDelivery: DateTime
}

# Represents a notification sent to a user
type Notification {
  id: ID!
  user: User!
  message: String!
  type: NotificationType!
  read: Boolean!
  createdAt: DateTime!
}

# Represents a user's profile information
type Profile {
  firstName: String!
  lastName: String!
  phoneNumber: String
  dateOfBirth: DateTime
}

# Represents an address
type Address {
  id: ID!
  street: String!
  city: String!
  state: String!
  country: String!
  zipCode: String!
}

# Represents a user's wishlist
type Wishlist {
  id: ID!
  user: User!
  # Paginated list of products in the wishlist
  products(first: Int, after: String): ProductConnection!
}

# Represents a user's shopping cart
type ShoppingCart {
  id: ID!
  user: User!
  # Paginated list of items in the cart
  items(first: Int, after: String): CartItemConnection!
  totalAmount: Float!
}

# Represents a payment method
type PaymentMethod {
  id: ID!
  type: PaymentMethodType!
  lastFourDigits: String
  expirationDate: DateTime
}

# Represents a user's membership plan
type MembershipPlan {
  id: ID!
  user: User!
  plan: MembershipPlanType!
  startDate: DateTime!
  endDate: DateTime
}

# Represents a user role with associated permissions
type Role {
  id: ID!
  name: String!
  permissions: [Permission!]!
}

# Represents a single permission
type Permission {
  id: ID!
  name: String!
  description: String
}

# Represents an image, typically associated with a product
type Image {
  id: ID!
  url: String!
  altText: String
}

# Represents a product attribute
type Attribute {
  id: ID!
  name: String!
  value: String!
}

# Represents an item in an order
type OrderItem {
  id: ID!
  product: Product!
  quantity: Int!
  price: Float!
}

# Represents a stock movement (inbound, outbound, or adjustment)
type StockMovement {
  id: ID!
  inventory: Inventory!
  quantity: Int!
  type: StockMovementType!
  createdAt: DateTime!
}

# Represents a response to a support ticket
type TicketResponse {
  id: ID!
  ticket: SupportTicket!
  responder: User!
  message: String!
  createdAt: DateTime!
}

# Represents an item in a shopping cart
type CartItem {
  id: ID!
  product: Product!
  quantity: Int!
}

# Represents a type of membership plan
type MembershipPlanType {
  id: ID!
  name: String!
  price: Float!
  duration: Int!
  features: [String!]!
}

# Connection Types
# These types enable pagination for lists of objects

type AddressConnection {
  edges: [AddressEdge!]!
  pageInfo: PageInfo!
}

type AddressEdge {
  node: Address!
  cursor: String!
}

type OrderConnection {
  edges: [OrderEdge!]!
  pageInfo: PageInfo!
}

type OrderEdge {
  node: Order!
  cursor: String!
}

type ReviewConnection {
  edges: [ReviewEdge!]!
  pageInfo: PageInfo!
}

type ReviewEdge {
  node: Review!
  cursor: String!
}

type NotificationConnection {
  edges: [NotificationEdge!]!
  pageInfo: PageInfo!
}

type NotificationEdge {
  node: Notification!
  cursor: String!
}

type PaymentMethodConnection {
  edges: [PaymentMethodEdge!]!
  pageInfo: PageInfo!
}

type PaymentMethodEdge {
  node: PaymentMethod!
  cursor: String!
}

type MembershipPlanConnection {
  edges: [MembershipPlanEdge!]!
  pageInfo: PageInfo!
}

type MembershipPlanEdge {
  node: MembershipPlan!
  cursor: String!
}

type ProductConnection {
  edges: [ProductEdge!]!
  pageInfo: PageInfo!
}

type ProductEdge {
  node: Product!
  cursor: String!
}

type ImageConnection {
  edges: [ImageEdge!]!
  pageInfo: PageInfo!
}

type ImageEdge {
  node: Image!
  cursor: String!
}

type AttributeConnection {
  edges: [AttributeEdge!]!
  pageInfo: PageInfo!
}

type AttributeEdge {
  node: Attribute!
  cursor: String!
}

type OrderItemConnection {
  edges: [OrderItemEdge!]!
  pageInfo: PageInfo!
}

type OrderItemEdge {
  node: OrderItem!
  cursor: String!
}

type PaymentTransactionConnection {
  edges: [PaymentTransactionEdge!]!
  pageInfo: PageInfo!
}

type PaymentTransactionEdge {
  node: PaymentTransaction!
  cursor: String!
}

type ShipmentConnection {
  edges: [ShipmentEdge!]!
  pageInfo: PageInfo!
}

type ShipmentEdge {
  node: Shipment!
  cursor: String!
}

type CategoryConnection {
  edges: [CategoryEdge!]!
  pageInfo: PageInfo!
}

type CategoryEdge {
  node: Category!
  cursor: String!
}

type StockMovementConnection {
  edges: [StockMovementEdge!]!
  pageInfo: PageInfo!
}

type StockMovementEdge {
  node: StockMovement!
  cursor: String!
}

type InventoryConnection {
  edges: [InventoryEdge!]!
  pageInfo: PageInfo!
}

type InventoryEdge {
  node: Inventory!
  cursor: String!
}

type TicketResponseConnection {
  edges: [TicketResponseEdge!]!
  pageInfo: PageInfo!
}

type TicketResponseEdge {
  node: TicketResponse!
  cursor: String!
}

type CartItemConnection {
  edges: [CartItemEdge!]!
  pageInfo: PageInfo!
}

type CartItemEdge {
  node: CartItem!
  cursor: String!
}

# Represents pagination information for connections
type PageInfo {
  hasNextPage: Boolean!
  hasPreviousPage: Boolean!
  startCursor: String
  endCursor: String
}

# Enums

# Represents the possible statuses of an order
enum OrderStatus {
  PENDING
  PROCESSING
  SHIPPED
  DELIVERED
  CANCELLED
}

# Represents the possible statuses of a support ticket
enum TicketStatus {
  OPEN
  IN_PROGRESS
  RESOLVED
  CLOSED
}

# Represents the possible statuses of a payment transaction
enum PaymentStatus {
  PENDING
  COMPLETED
  FAILED
  REFUNDED
}

# Represents the possible statuses of a shipment
enum ShipmentStatus {
  PROCESSING
  SHIPPED
  IN_TRANSIT
  DELIVERED
}

# Represents the types of notifications that can be sent to users
enum NotificationType {
  ORDER_UPDATE
  PRICE_DROP
  BACK_IN_STOCK
  PROMOTIONAL
}

# Represents the types of payment methods available
enum PaymentMethodType {
  CREDIT_CARD
  DEBIT_CARD
  PAYPAL
  BANK_TRANSFER
}

# Represents the types of stock movements
enum StockMovementType {
  INBOUND
  OUTBOUND
  ADJUSTMENT
}

# Input types for mutations

# Input for creating a new order
input CreateOrderInput {
  userId: ID!
  orderItems: [OrderItemInput!]!
  shippingAddressId: ID!
  billingAddressId: ID!
  paymentMethodId: ID!
}

# Input for specifying order items when creating an order
input OrderItemInput {
  productId: ID!
  quantity: Int!
}

# Input for updating a product's information
input UpdateProductInput {
  name: String
  description: String
  price: Float
  categoryId: ID
}

# Input for creating a new support ticket
input CreateSupportTicketInput {
  customerId: ID!
  subject: String!
  description: String!
}

# Input for filtering products in queries
input ProductFilterInput {
  categoryId: ID
  minPrice: Float
  maxPrice: Float
  inStock: Boolean
}

# Input for filtering orders in queries
input OrderFilterInput {
  status: OrderStatus
  minTotalAmount: Float
  maxTotalAmount: Float
  fromDate: DateTime
  toDate: DateTime
}

# Custom scalar for handling dates and times
scalar DateTime

First of all, this schema works. If you want to play with it, you can. You can go to this glitch server I hooked up that will actually run. If you send GraphQL requests to this endpoint, you'll get responses... and if you navigate there in your browser, you can play with the schema.

Next, I wanted to see if it could generate rails code in a D2 diagram. It took some coaxing, but I asked it to generate D2 nodes with ActiveRecord source code in them. Here's the model graph.

OK, what about the GraphQL layer?

Next, I asked Claude to imagine what a service layer might look like... I had a hard time getting Claude to do "bad" software engineering, so the service layer actually looks pretty reasonable to me.

Want to see what the whole monstrosity looks like?

Yep... that's a big ball of mud right there. That's a plate of Spaghetti. That's something that needs refactoring.

So why is this so cool? Beyond the fact that Claude's abilities are neat, it demonstrates that with an LLM we can make realistic, runnable synthetic code. Next, I'm going to try to use this code as a sort of lab for studying legacy enterprise codebases. Perhaps it will make it easier for us to share techniques across organizations and compare notes.

Off by one

Div III: Empowerment software

Introduction #

webGP #

What is GP #

What is Symbolic Regression #

Motivation #

Function choice and GP parameters #

Visualization of results #

Process of building #

Challenges #

What would come next #

Status #

ChemGP #

CarrotStick #

Trext #

Modfinder #

Opinion Essay #

Juggling data #

Time points #

Advocacy #

Advocacy #

Conclusion #

Await... async

Sequelize ERD

Installation #

API #

From bash #

Options #

We have coding in schools backwards

Offer scorecard

Pandemic writing practice

Using Vega Lite to explore chemistry

Playing with Hilbert

Cost of computing

Inflection points #

Plateaus #

"Earthquakes" #

What about the cost of writing programs? #

scrapbox #

stuff to weave in #

airtable evil? #

How to help a student get unstuck

Forget about the program, focus on the programmer #

Don’t let them overheat #

How to teach #

How to debug #

How to support #

Putting it all together #

Zeke's new brand rollout

New headshot #

New logo and name #

Why I refreshed my brand identity #

Engineering #

Content #

A word of caution on hiring rubrics

Semantic operation layer

What #

Why #

How would this look? #

Logging #

Why is Excalidraw so fucking good?

You're in #

You already know how to use it. #

Low fidelity #

Intuitive nod to pro tools #

Semantically oriented #

Collaborative with zero friction #

The takeaway #

Misc #

Helping students with pair programming

What is pair programming #

Does it work? Is it good for students? #

When things go wrong #

Preventing problems #

How to help #

Moving to a new job? You want a personal knowledge base

You're a knowledge worker. Build a knowledge base #

Plant seeds #

Eng wiki #

Introduction

webGP

What is GP

What is Symbolic Regression

Motivation

Function choice and GP parameters

Visualization of results

Process of building

Challenges

What would come next

Status

ChemGP

CarrotStick

Trext

Modfinder

Opinion Essay

Juggling data

Time points

Advocacy

Advocacy

Conclusion

Installation

API

From bash

Options

Inflection points

Plateaus

"Earthquakes"

What about the cost of writing programs?

scrapbox

stuff to weave in

airtable evil?

Forget about the program, focus on the programmer

Don’t let them overheat

How to teach

How to debug

How to support

Putting it all together

New headshot

New logo and name

Why I refreshed my brand identity

Engineering

Content

What

Why

How would this look?

Logging

You're in

You already know how to use it.

Low fidelity

Intuitive nod to pro tools

Semantically oriented

Collaborative with zero friction

The takeaway

Misc

What is pair programming

Does it work? Is it good for students?

When things go wrong

Preventing problems

How to help

You're a knowledge worker. Build a knowledge base

Plant seeds

Eng wiki

Github repos

People & Teams

Version tracking

Diagrams

Excalidraw

Mermaid

Questions

Daily notes

Tickets

Retros, free writing, and synthesis

I went for it

It's too small. Flipping the page is way too hard.

The eraser is kind of annoying

Shitty file system

No context or tie-ins

Bad sync

You can't really read books