Relay Game Update

Introduction

Ought is investigating ways of decomposing tasks into smaller sub-tasks. Task decomposition is not a new idea: it’s widely recognized as fundamental to modern economies. People have worked out ways to decompose complex tasks (e.g. create an electric car) into smaller sub-tasks (create an engine and battery, design steering controls, test the car for safety) which in turn are broken down into yet smaller tasks, and so on. The smallest sub-tasks are carried out by individual humans, who may have a limited understanding of how their work relates to the original task.

Our focus is on the decomposition of cognitive tasks, where the goal is to provide information or to answer a question -- without having to take physical actions. Cognitive tasks include solving a mathematics problem, analyzing a dataset, or summarizing a story. We are testing a bold hypothesis about the decomposability of cognitive tasks:

Factored Cognition Hypothesis: Any cognitive task can be decomposed recursively into self-contained, interpretable sub-tasks that can be solved by humans in ten minutes of work.^[1]

Sub-tasks being "self-contained" means they are solvable without knowledge of the task. If the task is a hard physics problem, then self-contained sub-tasks would be solvable for someone who hasn’t seen the physics problem (and need not know the task is about physics). This differs from most real-world examples of collaborative problem solving, where everyone in the team knows what task is being solved.

Testing the Factored Cognition Hypothesis involves the following steps:

• Find cognitive tasks that seem hard to decompose (e.g. because normally one person would spend days or weeks on the same problem rather than 10 minutes).
• Generate a high-level strategy that would plausibly decompose the task.
• Test whether the strategy from (2) works by having a group of people solve (1) under controlled conditions.

For some ideas on high-level strategies for a range of problems, see our Factored Cognition slides and the paper Supervising strong learners by amplifying weak experts (Appendix B). We are working towards experiments that encompass steps (1)-(3) and we have developed a web application, Mosaic, for solving tasks with groups of people. At some point we will report on experiments based on Mosaic. For now we describe some experiments with a simple form of Factored Cognition called the Relay Game.

In the Relay Game participants work on a task sequentially, with each person having ten minutes to help solve the problem before handing off to the next person (see Figure 1). This is similar to real-world situations where one person quits a project and someone else takes over. However, the big difference comes from the ten-minute time limit. If the task is complex, it might take ten minutes just to read and understand the task description. This means that most players in the relay won't have time to both understand the task and make a useful contribution. Instead players must solve sub-tasks that previous people have constructed.

We tested the Relay Game on programming problems from Project Euler. Here is a simple example problem:

How many different ways can one hundred be written as a sum of at least two positive integers?

Solving these problems requires both mathematical insight and a working implementation in code. The problems would take 20-90 minutes for one of our players working alone and we expected the relay approach to be substantially slower.

Experiment Design

Players worked on a shared Google doc (for notes) and code editor (see Figure 2). The first player receives only the Project Euler problem and begins making notes in the doc and writing code. After ten minutes, the second player takes over the doc and code editor. The Relay ends when a player computes the correct answer to the problem, which can be automatically verified at Project Euler.

We had 103 programmers volunteer to play Relay. They started 40 questions in total but only 25 had relay chains of more than 5 people. The total amount of work was 48 hours and only four questions were successfully solved. See Table 1 for a breakdown of a subset of the 40 questions. (Note: Most of the 103 players only worked on a few problems. Since much of the work was done by a smaller number of players, they were spread thinly over the 40 problems -- as each player spends just 10 minutes on a problem.)

Project Euler Problem	Relay Total Time	Solved by Relay?	Est. Solo Time to Solve
Prize Strings	30 mins	Y	27 mins
Counting Summations	40 mins	Y	24 mins
Reversible Numbers	70 mins	Y	30 mins
Palindromic Sums	100 mins	Y	60 mins
Blancmange	130 mins	N	35 mins
Leftovers	130 mins	N	35 mins
Anti-Chain	140 mins	N	35 mins
Prime Geometric	160 mins	N	35 mins
Prime Triplets	260 mins	N	35 mins
Pandigital	410 mins	N	80 mins

Can we conclude from Table 1 that Relay is much less efficient than the usual way of working on problems? Not really. It could be that Relay players would get better with experience by developing strategies for decomposing problems and coordinating work. So the main lesson from our experiment is that Relay with inexperienced players is probably less efficient at Project Euler problems. (We say “probably” because we did not conduct a rigorous comparison of Relay vs the usual way of solving problems).

Clickthrough Examples

We are interested in general failure modes for Factored Cognition with humans and in strategies for avoiding them. Our Relay experiment is a first step in this direction. We exhibit concrete examples from our Relay experiment that are suggestive of pitfalls and good practices for Factored Cognition.

Here are five “click-throughs”, which show how ideas and code evolved for particular Project Euler problems. In each case, the problem to be solved is shown on the bottom half of the Google doc on the first slide of the click-through.

Prize Strings

In these three attemps on Prize Strings the players quickly build on players previous work and get the correct answer.

Player One proposes a solution strategy for the problem and correctly identifies the recursive pattern.

Player two makes some minor modifications to the document and expands the code.

Player three tweaks the code, runs it, and gets the answer.

Empty Chairs

Empty Chairs was not solved but significant progress was made (with seven people contributing). The clickthrough demonstrates iterative improvements to a math heavy solution.

Player One starts by attempting to turn the problem statement into a formal specification.

Player Two adds more structure to the problem, commentary to the existing strategy, and defines the current task.

Player Three improves the first strategy and proposes a second solution strategy.

Player Four focuses on the "dynamic programming" strategy.

Player Five adds more structure and details to both of the existing strategies.

Player Six rejects the "dynamic programming" strategy and adds desiderata for any new strategy.

Player Seven first starts to code an answer.

Palindromic Sums

Palindromic Sums was solved in 10 attempts.

Player One creates an outline of the strategy and tasks for solving the problem.

Player Two adds more detail and formal structure to the strategy.

Player Three adds more detail to one step in the strategy and also starts implementing code.

Player Four adds a utility function.

Player Five notes a mathematical relationship in the problem, and links the two functions.

Player Six made no changes to the state of the problem.

Player Seven added logging to debug the speed of the solution.

Player Eight started to check the strategy and function logic.

Player Nine rewrote the code to use a generator syntax to improve performance.

The code was rerun with no changes but avoiding the timout issues, and the correct answer was generated.

A Scoop of Blancmange

There were 13 unique attempts on A Scoop of Blangmange. While technically unsolved, the answer was off by only 1e-8. Only attempts that changed the state of the problem are shown.

Player One proposed a strategy and wrote initial functions

Player Two pointed out a potential problem in the strategy and proposed a modified approach.

Player Three benchmarked the strategy.

Player Four tested the random_sample function.

Player Five tested the random_sample function.

Player Six proposed a new strategy focusing on a different paradigm using calculus.

Player Seven notes that current approach is too imprecise, and comments on improvements to the Reimann sum strategy.

Player Eight took the idea from the previous player and updated it.

Player Nine selected the strategy to use and added specific subtasks.

Player Thirteen (players 10-12 made no change to the problem state) refined the stategy and implementation.

Sum of a Square and a Cube

Sum of a Square and a Cube was unsolved with seven attempts.

Player One converted the problem into pseudocode, and listed next steps to take.

Player Two estimated the run time.

Player Three brainstormed ways to speed up the code.

Player Four made a small note but was probably cutoff from finishing.

Player Five added to the pseudocode solution.

Player Six brainstormed more ways of speeding up the code.

Player Seven evaluated which of the potential solutions was most promising.

500

With five attempts 500 went unsolved, but demonstrated interesting use of metadata tags.

Player One simplified the problem and started generating tasks.

Player Two analyzed the mathematical properties of the problem.

Player Three started analyzing the primality of the target number (but likely ran out of time before writing anything).

Player Four determined that 500500507 was not prime, and added to next steps and the code solution.

Player Five proposed a tactic for generating the multiplication table.