Agile, Lambdas, and Validation

Logistics

If you haven't yet installed Node or cloned the starter code, it would be a good idea to do that ASAP.

We don't have a no-laptops policy in 0320. But if you're going to play chess or write an essay while in class, please sit where you won't distract anyone sitting behind you.

Agile Development

We're going to be using a few terms of art in 0320. While we do put a bit of our own spin on these (largely because this is a class, and not a full-time job) it's important to understand what's going on.

We say that a development methodology is agile if, broadly speaking, it prioritizes the ability to change plans in response to regularly-sought feedback. Once we try to define it more precisely, you'll find multiple competing definitions, and many competing methodologies that purport to be "agile". But change and feedback are good enough for us today.

Usually, you'll see the term used in contrast to "waterfall" development, where the software project proceeds along a linear path, like this:

• Requirements
• Specifications
• Design
• Implementation
• Testing
• Maintenance

Of course, all of these "phases" still exist in an agile project! The difference is that (e.g.) customer feedback on an early demo might result in changes to the requirements, or some trouble with testing an early prototype might mean changing the design to make testing easier. To make this possible, agile projects usually start development early, expecting that some, or even most, of that prototype code will be replaced later. But something must be possible to demo at every stage, or feedback is hard to obtain.

Sprints

Agile development is often divided into short periods of development effort: sprints. The duration of a sprint varies depending on the project and company. In 0320, we've organized development into one-week sprints with relatively small requirements for each. Each sprint builds on the last.

Organizing sprints is an important skill for project managers. Timelines need estimates (and the ability to alter those estimates when needed), tickets need to be managed, and so on. We'll gloss over most of this in 0320, although your term project will need some attention to project-management tasks like these.

User Stories

A user story is a short description of desired behavior. You'll often find these used in requirements documents for agile projects. There are a few templates, but we (mostly) follow the "As a USER-ROLE I can NEEDED-BEHAVIOR so that TASK-OR-BENEFIT" pattern. Look at the user stories in sprint 1.1, and you should see this pattern.

It's important to remember that, although they follow a template, user stories are informal: they may not be enough by themselves to really describe what the customer needs. To help bridge this gap, they are often accompanied by acceptance criteria, which give additional lower-level requirements. But these are still fairly informal, and there's often a need for more precise specification to be agreed upon between developers!

But what makes a good user story? A user story describes a narrow, demoable piece of functionality that can reasonably be accomplished in a single sprint. Often, you might start with a rough user story and realize that it's too big, and then split it into multiple stories. If users request an enhancement to an existing story, commonly these would be added as new stories (and more infrequently as additional acceptance criteria).

Reading: Clever Manka

There's a fairy tale I like: Clever Manka, from a 1920's compilation called The Shoemaker's Apron, available on Project Gutenburg here. I'll reproduce the text in full here, within these spoiler tags.

Stories like this appear in numerous places throughout the world. For example, India has the story of Hiranyakashipu, a king with a boon protecting him from death "by day or by night", "indoors and out", etc. Narasimha, an avatar of Vishnu, finds a way around the riddle.

These stories are so common that I might say humans find something universally valuable about them. but what in the world does a fairy tale have to do with testing?

Don't worry about whether your answer is the same as mine. Especially in 0320, there are often many different good answers, even to technical questions.

Boundary Conditions: Testing and Defensive Programming

Here's an example: "I have tested a positive number, and I have tested a negative number." Surely all numbers are positive or negative. (Is this true?)

Here's another example: "I have tested this function, which accepts a Java Boolean, on both values: true and false." (What's missing?)

Never assume that the obvious partition of the space actually covers the space; be on the lookout for special cases, outliers, and even new dimensions about which to think. This isn't only about testing, either---see, for example, the Falsehoods Programmers Believe About Time. Part of programming defensively is trying to avoid making unnecessary assumptions, while still allowing for extensibility. Keep this in mind as we start to code (and test) together.

Runtime Validation

Ok, so we're on the lookout for faulty assumptions that we might make when programming. But what about other people who might be writing code we depend on—or even code that calls ours? We can't solve this problem, and (as careful as we might be) we can't entirely solve it for ourselves, either. So we'll need to make our code robust against bugs, whether they are our mistakes or others'.

Runtime validation involves checking for issues with data your code is given, unexpected changes in state, and so on. Here are two examples.

Example 1: User Input

Suppose that a user just typed in a pair of numbers and we want to know if one is bigger than another. Because they just arrived, they will be strings, so we need to convert them. No problem:

function rawGreaterThan(arg1: string, arg2: string): boolean {
    return parseInt(arg1) > parseInt(arg2)
}

This seems reasonable enough, but it forgets something important about numbers in JavaScript: NaN (not a number) is a number. Since TypeScript is JavaScript with added protection, the same is true in TypeScript. Imagine that a user accidentally types "!00" instead of "100". In Java, we'd get an exception since "!00" can't be converted to a number. But in TypeScript, parseInt("!00") produces a value: NaN. And NaN isn't greater or less than anything! (It can't even be equal to itself: NaN == NaN evaluates to false.) So:

> parseInt("!00") > parseInt("100")
false
> parseInt("!00") <= parseInt("100")
false

Oh, dear. It's one of those pesky boundary conditions! But the problem is invisible outside of our function. The caller only gets a boolean. Let's do some validation to protect them and us. We'll first check that both arguments can be converted:

function rawGreaterThan(arg1: string, arg2: string): boolean | undefined {
    const num1 = parseInt(arg1)
    const num2 = parseInt(arg2)
    // num1 === NaN won't work! Instead:
    if(Number.isNaN(num1) || Number.isNaN(num2)) return undefined
    return parseInt(arg1) > parseInt(arg2)
}

Now the caller will be given something special if there's an issue converting the input to numbers. We could improve this, actually: undefined is better than hiding the problem, but it doesn't communicate much. We could throw an error, but instead let's explore building a better error value.

interface ConversionError {
    error: "parseInt"
    arg1: string 
    arg2: string
}

Yeah, that's a string literal as the type of the error field. In TypeScript, this means the type containing only that string. Thus, the error field can only ever be the string "parseInt" So why have that field at all? Because if our larger program can ever produce a different kind of conversion error, we can add other options via union types:

interface ConversionError {
    error: "parseInt" | "parseFloat"  // | ... | ... | etc.
    arg1: string 
    arg2: string
}

If we said error: string instead, we'd be allowing any old string. Sometimes this is what you want, but if you're establishing a protocol for reporting errors, you probably want an enumeration of error codes, not arbitrary strings. Anyway, now we can write:

function rawGreaterThan(arg1: string, arg2: string): boolean | ConversionError {
    const num1 = parseInt(arg1)
    const num2 = parseInt(arg2)
    if(Number.isNaN(num1) || Number.isNaN(num2)) 
        return {error: "parseInt", arg1: arg1, arg2: arg2}
    return parseInt(arg1) > parseInt(arg2)
}

The caller is now being told exactly what the problem is, not only that they had some kind of problem. This is only possible because we're now validating the input.

Reading Data

The same problem occurs if you're reading in data from files or responses to your web requests. Suppose the registrar stored student information in a comma-separated-value (CSV) file:

Name,Credits,Email
Tim Nelson,10,Tim_Nelson@brown.edu
Nim Telson,11,MYAWESOMEEMAIL

If I expect the third column to contain an email address, I'm going to be very surprised with the data I get from the second row. Hopefully the registrar is validating every column of every row so they can avoid relying on the bad data. But now it's not as simple as it was before: do I need to manually write a validation function that matches all possible email addresses? Argh!

Library Support: Zod

Fortunately, people write and share libraries that solve real problems. We'll be using a library called Zod, which is built to help us with exactly these kinds of validation tasks. Zod is free, open source, and widely used. Zod works off of something called a schema, which is kind of analogous to a type (not entirely; more on this later). Here's a schema to validate each CSV row according to the registrar's expectations:

const studentRowSchema = z.tuple([z.string(), z.coerce.number(), z.email()])

Zod schemas are compositional. z.string() is a schema that matches any string. z.coerce.number() is a schema that matches any number or any string that can be converted to a number. z.email() is a schema that matches strings containing email addresses. And z.tuple takes an array of schemas and matches arrays of exactly the same length, where each of the sub-schemas matches its corresponding array element.

I hear that at this point, you might be enhancing a CSV parser. Probably it splits a big file into rows, and then splits each row by , (or something like that) to produce an array. So for the 2-row example above, you might get a string[][] that looks like:

[["Tim Nelson","10","Tim_Nelson@brown.edu"],
 ["Nim Telson","11","MYAWESOMEEMAIL"]]

The first row will match the schema, but the second won't: "MYAWESOMEEWMAIL" isn't an email address. Zod will give a structured error for this failure, although it contains even more detail than the parseInt error we built before.

Zod makes validating external data much, much easier. Whenever validation becomes non-trivial, stop writing ad-hoc if statements and use Zod instead.

Anonymous Functions

Zod doesn't just do validation: it can also transform data into new shapes. But it needs us to tell it how that transformation works: a strategy function that says how to map the old shape into the new one. Often, we'll use an anonymous function for this. Anonymous functions are sometimes called "lambdas" (e.g., in Python). Even Java has them in the form of the Function<T, R> interface and special syntax to build them concisely. TypeScript's syntax for these is quite similar to Java's.

For example, let's turn a (validated) CSV row into an object instead:

const studentRowSchema = z.tuple([z.string(), z.number(), z.email()])
                          .transform(arr => ({name: arr[0], credits: arr[1], email: arr[2]}))

Now the first row becomes an object: {name: "Tim Nelson", credits: 10, email: "Tim_Nelson@Brown.edu"}.

Passing functions arguments to to other functions is so powerful that it appears in multiple contexts. In Object-Oriented Programming, you see it everywhere under the name of "strategy pattern". For example, Java's Collections.sort method takes an object called a Comparator. A Comparator implements a method that takes two elements of the collection and says whether one is greater than another. In this way, the Collections library allows a single type to be sorted many different ways.

Exercise

I'm not going to start the required exercises this early in shopping period. But I'd still like us to get some practice with TypeScript.

Tools Challenge: package.json

Let's start with the toolchain we're using in the course. You've probably cloned the starter repository by now. Let's look at the package.json file together. JSON means "JavaScript Object Notation", and it's a very common text data format. You'll see fields like dependencies and scripts and so on. What do you think they mean? How do they interact with the npm console command?

Design Challenge: Unchecked Casting

Just like Java, TypeScript has a way to tell the type checker to be quiet because you know best. We call these "unchecked typecasts". In TypeScript, this happens whenever a value has type any. You might explicitly cast this (via as any) or TypeScript might infer the type because it has no further context. The any type exists because TypeScript needs to interoperate with untyped JavaScript, and it's dangerous to keep around if you don't need it.

Union types mean that TypeScript may be "uncertain" about your code. Here's an example:

function whatToDo(input: string[] | number): string {
    return input[0]
}

If the input is a string array, this is fine. But what if it's a number? TypeScript reports this problem with the following error:

Element implicitly has an 'any' type because expression of type '0' can't be used to index type 'number | string[]'.
  Property '0' does not exist on type 'number | string[]'.ts(7053)

What should you do about this?

function whatToDo(input: string[] | number): string {
    if(typeof input === "number") return ""
    return input[0]
}

We'll talk more about this trick soon.