Eamonn O'Brien-Strain

MastodonBlueskyThreads

States with highest per-capita Covid death rates

Some highlights from today's graphs of per-capita Covid death-rates:

  1. In the ranking of countries, Latin America is doing worst, taking up the top seven spots. The United States is not far behind in the middle of the pack, with the United Kingdom and Sweden not doing much better. The rest of Europe is now doing pretty well, which is quite a turnaround from earlier in the pandemic.
  2. In the US, looking at the ranking by state or by county shows what has been much reported in the news, that states which escaped the initial impact of the pandemic now have the highest ranking, lead by Arizona and Mississippi. But some of the states initially hit hard, particularly New Jersey are also seeing high death rates again. New York is now doing quite well, at the bottom of the ranking, but California, which had done well in the past is now climbing up worryingly out of the middle of the pack.
  3. In the ranking of California counties, Imperial county is still by far the worst affected, but this is off a very small population base. Of the counties with large populations there is a big divide between NorCal and SoCal, with the southern counties all in the high up in per-capita deaths and the Bay Area counties all low down.

Oh yes, in an ideal world we would use a debugger to understand what our code is doing, but setting up a development environment to allow debugging can sometimes be tricky or annoying to do.

So often we resort to the venerable art of “printf debugging”: adding judiciously placed print statements in our code.

In JavaScript this means simply adding console.log statements.

But if you are writing code in a functional style, it can sometimes mess up your code to add print statements.

For example imagine you have a bug in some code that looks like this ...

const dotProd = (v1, v2) =>
  zip(v1, v2).map(pair => pair[0] * pair[1]).reduce((a, x) => a + x, 0)

... and you want to inspect the value of a + x returned by the reducer.

To use console.log you would have to rewrite the reducer to this inelegant form:

const dotProd = (v1, v2) =>
  zip(v1, v2).map(pair => pair[0] * pair[1]).reduce((a, x) => {
    const result = a + x
    console.log('Reducer returns', result)
    return result
  }, 0)

This really makes it inconvenient and error-prone to add these temporary logging statements, and to remove them when you're done.

So I published the pp function as a more elegant alternative.

To use it in Node JS, install with

npm install passprint

and then import it into your JavaScript file like so:

const { pp } = require('passprint')

and then you can do

const dotProd = (v1, v2) =>
  zip(v1, v2).map(pair => pair[0] * pair[1]).reduce((a, x) => pp(a + x), 0)

Note the additional pp(...) near the end. This is much less ugly than using console.log.

This will log out lines that look something like:

|||||||||MyClass.myFunction myFile.js:46:63 22ms 78878

where 22ms is the time elapsed since logging started and 78878 is the value of a + x. The number of | characters shows the depth of the call stack.

The pp simply returns the value passed to it, so it can be used with minimal perturbation to your code. It is easy to add and remove.

You don't need any extra disambiguating message with pp because the log messages include the function name, file name, and line number where it was invoked from.

More details in the README on Github.

Feel free to give feedback on Twitter or by adding a GitHub issue.


Old vs. new design for this blog

[Edit 2023-02-05 – this post is referring to a previous instantiation of this blog]

I just updated the visual design of this site to be a little more interesting than the default theme I had been using since I ported the site to Jekyll. I decided to tweak the CSS to get a neumorphic effect, which is similar to Material Design except instead of things floating above the surface they are indented into or stick above the surface.

It's pretty easy to get the effect with a CSS box-shadow, and thanks to @adam_giebl there's a handy online neumorphism generator to get you started.

I also decided to give this blog a name. “Yak Shavings” seems appropriate, since a lot of the technical things I write about here are the result of me happily going off on repeatedly nested tangents as I noodle on my side projects.


Bajel

I’ve been programming on Unix and Linux machines for many decades now, and one universal I’ve leaned on is the default availability of make, which has been labor-saving swiss-army knife for setting up my build and development flows, or for organizing any random commands that I execute repeatedly in a directory. It’s a great memory aid, reminding me when I come back later of how to work with the files in that directory.

Here are some examples of Makefiles I created a decade ago:

  1. When I was doing Palm webOS development I wrote a Makefile that has an eclectic collection of commands that comprised my hacked-together Palm build system.
  2. When I was doing Java web-server development, I wrote a blog post explaining how I used a Makefile to integrate the cool continuous-integration system of the day (Hudson) with the cool Java web framework of the day (Play Framework).

The things I like about make is that:

  1. It has very little boilerplate.
  2. It captures the commands, just as you type them in the command line.
  3. It captures the dependencies between the commands.
  4. The dependencies are simply the files generated by one command that are used by another command.
  5. The files can be file patterns, so you can express how one type of file is generated from another in general terms.
  6. It uses the timestamps of input and output files of the commands to determine if a command is out of date and needs to be run.
  7. It has variables to factor out common strings.
  8. It is ubiquitous and installed by default everywhere.

But I have been becoming increasingly dissatisfied with make. It was fine-tuned for the development environment of its day (compiling C on UNIX), and though it is remarkably versatile it still has lots of vestiges of special build-in support for that development flow. And over the decades it has accumulated a lot of complexity in how patterns and variables work.

However, I never found any replacement system that has all the positive features of make. The nearest is Bazel which is a multi-repo public version of Google’s internal monorepo build tool (which replaced an earlier make-driven build system in Google).

So I built my own tool, Bajel.

These days, most of my side projects use the npm ecosystem, so I built Bajel there. If you have npm installed, you can execute Bajel simply by doing

npx bajel

Bajel expects that the current directory contains a build file which in its most straightforward form is a JSON file. However Bajel also supports build files in other syntaxes, including YAML or TOML syntax, which make for cleaner files.

Here is an example build.yaml file, which is probably the closest of the available syntaxes to a classic Makefile:


TEST: ava test/contract_test.js

serve:
    deps:
        - dist/index.cjs
    exec: python -m SimpleHTTPServer 8888

test:
    deps:
        - test_default
        - test_contract_production
        - test_contract_development
        - test_contract_no_env

test_default:
    deps:
        - dist/index.cjs
    exec: ava

test_contract_production:
    exec: NODE_ENV=production $(TEST)

test_contract_development:
    exec: NODE_ENV=development $(TEST)

test_contract_no_env:
    exec: NODE_ENV=  $(TEST)

"perf.csv":
    deps:
        - src/node/perf.js", "src/common/optimizer.js"]
    exec: node $<

"dist/index.cjs":
    deps:
        - rollup.config.js
        - src/node/index.js
        - src/common/index.js
        - src/common/random.js
        - src/common/color.js
        - src/common/optimizer.js
        - src/common/contract.js
        - src/common/random.js
    exec: rollup --config $<

publish:
    deps:
        - dist/index.cjs
    exec: npm publish

clean:
    exec: rm -rf dist

However YAML as a syntax has fallen out of favor in some quarters, so you might prefer the following, which is semantically identical, but in the nice clean TOML format.


TEST="ava test/contract_test.js"

[serve]
deps = ["dist/index.cjs"]
exec = "python -m SimpleHTTPServer 8888"

[test]
deps = [
    "test_default",
    "test_contract_production",
    "test_contract_development",
    "test_contract_no_env",
]

[test_default]
deps = ["dist/index.cjs"]
exec = "ava"

[test_contract_production]
exec = "NODE_ENV=production $(TEST)"

[test_contract_development]
exec = "NODE_ENV=development $(TEST)"

[test_contract_no_env]
exec = "NODE_ENV=  $(TEST)"

["perf.csv"]
deps = ["src/node/perf.js", "src/common/optimizer.js"]
exec = "node $<"

["dist/index.cjs"]
deps = [
    "rollup.config.js",
    "src/node/index.js",
    "src/common/index.js",
    "src/common/random.js",
    "src/common/color.js",
    "src/common/optimizer.js",
    "src/common/contract.js",
    "src/common/random.js",
]
exec = "rollup --config $<"

[publish]
deps = ["dist/index.cjs"]
exec = "npm publish"

[clean]
exec = "rm -rf dist"

Some features to note in the above:

  • TEST="ava test/contract_test.js" is setting a variable which is referred to later in the file as $(TEST). In TOML syntax all the variables have to be defined at the top of the file before anything else.
  • [serve] is the first target, and is the one whose action is executed by default when you do npx bajel. You can execute any other target by adding it as an argument to the command line, for example npx bajel test.
  • deps = ["dist/index.cjs"] says that serve has one dependency, the target dist/index.cjs. If that file does not exist or is older than any of its dependencies, then the action for dist/index.cjs is executed before serve is executed.
  • exec = "python -m SimpleHTTPServer 8888" specifies the shell command that executes for serve. This is executed just as if you had typed python -m SimpleHTTPServer 8888 on the command line.
  • exec = "rollup --config $<" contains $< which is replaced by the first dependency, in this case rollup.config.js. Also possible are $+ which is replaced by all the dependencies (blank-separated), and $@ which is replaced by the target.

Bajel implements all the features of make that I value (including % pattern patching not shown in this example), but keeps things as simple as possible.

For advanced use, there is also a JavaScript syntax, which allows the actions to be specified as JavaScript functions, making each target be like a cell in a spreadsheet. There’s also a markdown format, to allow for literate programming of build files.

For details, see the README on Github.

If you have any feedback, feel free to add a GitHub issue, or reach out to me on Twitter.

Naming note: “Bajel” is Bazel, but with a “j” for JavaScript. It is pronounced /ba-hel/ in the Spanish fashion, as it is a Spanish word for a sailing ship such the Santa Ana, which is pictured at the top of this post.


Smoothish

[Updated 5-May-2020 to change the examples to the new 1.0.0 API.]

I published a new npm module called smoothish that smooths out time-series data without some of the drawbacks of the usual moving-point average.

When working on the visualization of per-capita COVID-19 death rates I needed a way to smooth out the curves of some noisy and incomplete data, and I wanted the data to extend up to the most recent day.

Standard moving-average did not meet those requirements, so I ended up writing my own smoothing function which you can use like a moving average, but it does not drop the points at the beginning or (more importantly) at the end.

It works by, at every point, doing a least-squares linear interpolation of other points nearby. It is flexible enough to handle missing points or points at the boundaries.

It's easy to use. In your JavaScript project install it with:

npm install smoothish

Then using it is as simple as:

const smoothish = require('smoothish')

const daysPerMonth = [
    31, 28, undefined, 30, 31, null, 31, 31, null, 31, 30, 31]

smoothish(daysPerMonth)
// --> [ 30.0, 29.4, 29.8, 30.1, 30.5, 30.6, 30.8, 30.8, 30.8, 30.7, 30.6, 30.7 ]

Note, that not only does it produce a smoothed output, but it also interpolates missing values.

By default the function uses a radius of 2, indicating the width of the neighborhood of other points to be considered. All points are actually considered by default, but with the ones closer having more weight falling off exponentially with a time constant of the radius.

The smoothish functions also has options to do moving average and to have a step-function falloff, just like a normal moving average. In that case a radius of 2 would specify a five-point moving average.

The smoothish function always returns the same number of output values as in the input.


Graph of COVID-19 Growth

To make it easier to compare how COVID-19 is affecting different countries I created these population-adjusted graphs.


standard JS logo

War is over, if you want it

War is over now

— Lennon

Have you been a belligerent in the code-formatting wars? Do you have strong opinions on tabs vs. spaces or on the merits of The One True Brace Style over other indentation styles?

Something has now changed that renders these wars obsolete.

When I joined Google one culture shock was discovering how picky Google was about seemingly trivial details of formatting. Time and again my code reviewers objected to what I thought was my creative expression of coding craft. They were unmoved by my assertion that b*b - 4*a*c was more readable than b * b - 4 * a * c or that FOO was a better name for a constant than kFoo.

Quickly I came to peace with the style guides and recognized their value. I realized that operating in Google's fabled monorepo I was reading more code than I was writing, making me appreciative of how a consistent style made the code much easier to read. Also as a code reviewer and tech lead I appreciated how just a link to a section of the quasi-legal style guides could cut short tedious bikeshedding on formatting issues and allow us to concentrate on the important issues.

Initially it was a bit tedious getting all the spacing right and remembering all the subtle line-breaking rules.

But over time it got better. First the linting tools improved and covered more of the style guide so that I generally could find and fix all the issues before sending code changes for review.

But then it got even better, with the linting tools supplemented by tools that automatically reformatted code according to the style guide for that language. Now I just basically no longer spend any brain cycles on formatting: I write arbitrary sloppily-formatted code and have it fixed up when I save the file.

The designers of the Go language were Googlers, and they bought into this philosophy. Pretty much all Go code is formatted by the gofmt program, which always reformats to the same standard style.

So when I started a personal JavaScript project I really wanted a JavaScript equivalent to gofmt which enforces a standard style.

In the JavaScript ecosystem, Eslint is the standard linter, which not only can warn against style violations, but can automatically fix them. And what's more it can go beyond simple indentation and spacing and fix more complex rules (all the ones marked with a 🔧 wrench in the list of rules).

Initially I assumed I would want to use the Google JavaScript style guide and the eslint rules that enforce it. I configured Eslint with

 {
    "extends": [
        "eslint:recommended",
        "google"
    ],
    "env": {
        "browser": true,
        "node": true
    },
    "parser": "esprima",
    "parserOptions": {
        "ecmaVersion": 6,
        "sourceType": "module"
    }
}

However, it was a bit unsatisfying to have a non-trivial config file with various things that need deciding and tweaking. It's also a bit annoying that it cannot not autofix all the problems it finds, so you still have to manually fix some of them.

Nevertheless it works fine, and produces nicely formatted code like below in standard Google format.

export const newCards = () => {
  const cards = [];
  for (let i = 0; i < 2; ++i) {
    const reversed = (i === 1);
    forEachPhrase((phrase) => {
      const responses = [];
      cards.push({phrase, reversed, responses});
    });
  }
  return cards;
};

But then I discovered standard JS. This has everything a auto style formatter should have. It is really easy to install, has good VS Code support, requires no configuration, and can fix all the problems it finds. It produces code like this:

export const newCards = () => {
  const cards = []
  for (let i = 0; i < 2; ++i) {
    const reversed = (i === 1)
    forEachPhrase((phrase) => {
      const responses = []
      cards.push({ phrase, reversed, responses })
    })
  }
  return cards
}

The most notable difference is that it omits semicolons.

Omitting semicolons is something the JavaScript language allows, but the exact rules for when you can do so have some subtle corner cases that can cause ambiguity and unexpected interpretation of code, so the conventional wisdom has long been to always include semicolons. That's the view of the Google style guide.

But the standard JS community have decided that the cleaner semicolonless look is preferred and have tackled the corner cases (which are rare) by making sure they are handled by other standard JS rules.

Personally, I don't care whether there are semicolons or not, but I'm going to stick with using standard JS for my projects as the tooling is better and it looks like it's gaining momentum in the JavaScript community.

Now of course pretty-printing has existed for a long time. The indent formatter dates from 1976, while text editors and IDEs have long been able to autoformat. But all these pretty-printers have one fatal flaw: they are too flexible, and can be configured with different style options.

The revolutionary novelty of Go's gofmt, standard JS, and Google's internal tool is that they have no options. They enforce a single standard style.

The wars are over! Everyone should just use a standard auto-formatter with no options.


Flash Card

There were some interesting aspects to building this web app.

  1. The spaced repetition algorithm for displaying flash cards to maximise learning.
  2. Using locale-specific web search to fetch the images
  3. Architecting the app as a static web site that uses local storage to store user state.
  4. Using vanilla JavaScript with no framework.
  5. Using 3D animations for flipping cards.
  6. Custom creating all the CSS styling.

In this post, I'll discuss the spaced-repetition algorithm, and defer the other aspects to future posts.

From Wikipedia

Spaced repetition is an evidence-based learning technique that is usually performed with flashcards. Newly introduced and more difficult flashcards are shown more frequently while older and less difficult flashcards are shown less frequently in order to exploit the psychological spacing effect. The use of spaced repetition has been shown to increase rate of learning.

There is a variety of published research on spaced-repetition, and different flashcard apps use a variety of different algorithms, some quite complicated.

I decided to start with something much simpler. Every time a user sees the card they say how accurately they predicted the reverse side of the card. We store a response object that has this correctness score along timestmp t of the response. For each card we calculate an average score, but weighted by time so that more recent scores count for more. Then we sort the list of cards so that the cards with the lowest scores come first.

The core of the algorithm is in common.js:

const TAO = 1000.0 * 60 * 60 * 24

export const score = (responses) => {
  if (responses.length === 0) {
    return -1
  }
  const now = Date.now()
  const weightedSum = responses
    .map((response) =>
         response.correctness * Math.exp((response.t - now) / TAO))
    .reduce((sum, x) => sum + x)
  return weightedSum / responses.length
}

The score function goes through the array of responses and sums up the correctness values weighted by an exponential decay function with a time constant TAO of 24 hours. This means, for example, that a day-old response is weighted at about 0.37 of a recent response.

This score is called from the sort function in index.js:

  const sort = () => {
    cards.sort((a, b) => score(a.responses) - score(b.responses))
  }

My hypothesis is that this simple algorithm with its single TAO parameter is as effective as the more complex algorithms. Hopefully by tuning TAO I can emulate those algorithms.

An observant reader will have noticed some significant algorithmic inefficiencies in the above naive implementation of the algorithm. In particular there are some superlinear big-O complexities that I'll probably need to mitigate as I add more cards to the app.


If you are reading this article on a phone or other touch device, did you know you can swipe left and right to get to other articles? Go on, try it now. I'll wait.

[Edit 2023-02-07: Actually you will need to try this on the old version of this blog.]

This turned out to be easy to implement thanks to Hammer.js. Here's how I did it.

  1. I downloaded the Hammer.js library into my JavaScript directory:
   wget https://hammerjs.github.io/dist/hammer.min.js

and creates a new file called swipe.js in that directory. 2. I added the following to the HTML template of my article pages:

   <script src="/js/hammer.min.js" defer></script>
   <script src="/js/swipe.js" defer></script>

This loads the Hammer library and my application code into the web page. As an optimization I add the defer attribute so that the script loading does not block the initial page load: after all this is optional stuff that should not slow sown the normal functionality of the page. 3. I already had the following navigational structure in the article pages for going to the previous and next pages:

   <nav>
       <a id="prev" href="...">&laquo; ...</a>
       <a id="next" href="...">... &raquo;</a>
   </nav>

The JavaScript can pluck out the destination URLs from the href attributes of the #t and #prev elements. 4. I added the following to swipe.js:

   const mc = new Hammer(document.body)

   mc.on('swipe', (ev) => {
     const dir = (ev.deltaX < 0) ? 'next' : 'prev'
     const a = document.getElementById(dir)
     if (a) {
       window.location = a.href
     }
   })

This code initializes Hammer.js and sets up a swipe lister. When a swipe is detected the code looks at the deltaX event property to determine if this was a left or right swipe, pulls out the URL from the href of the corresponding link, and navigates to that link.

This simple solution works fine, but there is no feedback to the user to show that a swipe has happened. In the next article I'll show how to add animation to improve the UX.


In a follow-up to the last article which described Adding swipe actions to a web site, I'll show how to add simple animation to give some user feedback that the swipe has happened.

First of all, we'll modify the JavaScript to add a single line as follows:

mc.on('swipe', (ev) => {
  const dir = (ev.deltaX < 0) ? 'next' : 'prev'
  const a = document.getElementById(dir)
  if (a) {

    // Added line for animation:
    document.body.classList.add('animate-' + dir)

    window.location = a.href
  }
})

So, just before loading in the new page this code adds a CSS class animate-next or animate-prev to the body element of the page (i.e. to the entire page). This will trigger an animation that will run while waiting for the new page to load.

To actually cause the animation we add the following CSS:

  @keyframes slideLeft {
    to {
      transform: translateX(100%);
      opacity: 0;
    }
  }
  @keyframes slideRight {
    to {
      transform: translateX(-100%);
      opacity: 0;
    }
  }
  .animate-prev {
    animation: 0.2s ease-out 0s 1 slideLeft;
  }
  .animate-next {
    animation: 0.2s ease-out 0s 1 slideRight;
  }

Let's walk through this:

  1. Each @keyframes includes a single to (100%) key-frame which is the state to animate to. Because there is no from (0%) key-frame the animation will default to using the current state as the starting state. It
    • translates 100% to the left or right (which for the body will translate it offscreen)
    • fades from opaque to transparent.
  2. Each class selector has an animation that uses the corresponding key-frame, specifying that it should animate
    • over the given time
    • using ease-out (start fast then slow down)
    • with no delay
    • playing just once
    • the given #keyframes name

Note we are careful to only use High Performance Animations that can be efficiently implemented by a phone's GPU, namely translation, scaling, rotation, or opacity change. We could obviously use these four types of animations to further enhance the page-swiping animation to make it more effective, while still keeping it efficient.