<Squalltalk center={[-61.81616, 17.07385]} zoom={8}/>

Jamaica Kincaid's A Small Place is a Caribbean Everyman. In the sense that it represents the anguish of a discarded people. In the way that it demonizes the tourist class of whites who yearly inundates the jeweled islands of the Antilles. The accusation and indemnity leveled at mopey Moby Dicks and jowled Jane Fondas is Kincaid's. It also belongs to slave descendants who sing commercialized welcomes to cruise patrons whom they curse under their breath. It is hardly surprising that a country whose success in inextricably tied to ours is resentful, while we have disdainful regard for a people barely flourishing on our excess wealth. Our ideal of paradise is incompatible with the possibility of suffering. Escape cannot reconcile with the effort of involvement. The romanticism of the virgin islands is rather spoilt by the recollection of their rape. A harsh analogy: some cultures with backward notion of women's right force women into marrying their rapists in an effort to rid the family of damaged goods. So are the tropical maiden islands wed to rapacious and sinful world powers who exploited the beauty and naivety to launder bloody monies and escape the pollution and clamor of first world civilization.

The children of this union must decide between the lure of prosperity in the first world, or commitment to (and likely impoverishment in) the mother country. Hybrids are possible, but at the cost of integrity. Kincaid speaks for her home, but is an ex-patriot. Her neighbors know her and her love of Antigua. We know her only for her enmity to her neighbors. I wonder how her neighbors feel when they read her barrage against vacationing Americans. I am not shocked. The same language enters the world across white tongues as it does by black tongues. How often do we hear demands that tourists in American learn english, and not bother us in broken imitations of our gilded language? How often do Americans laugh at gawking foreigners, their raised cameras, or their willingness to pay exorbitant prices for local foods or activities which may or may not have any root in our history or culture.

The Antiguan who has never left the island and yet condemns visitors is another matter, for how can she possibly know the modern inconveniences we contend with every day? Surely slaving away continually to provide food and shelter pales in comparison to the injustice of paying bank fees. Knowing a suburban marijuana dealer is certainly as scandalous as knowing someone who was killed by a cigarette boat drug smuggler from Central America. The financial burden of state imposed health care must be worse than not having health care because you can't afford it, and in any case have no hospital worth sending your loved ones to. Any money we spend in Antigua is a luxury. Milk and gasoline at twice the continental price is a novelty, not a curse. 

American education denounces generalization. This is problematic, because of course it espouses equality equally. We long ago decided that enslavement is immoral, our ancestors even fought and died over this. Christians, and anyone vaguely familiar with Christ, know that one must forgive and forget. How absurd that missionaries concerned (possibly) with the souls of savages, paved the way for colonialism through indoctrination, and left them with core values of forgiveness and righteous suffering. Caribbean cultural resentment is experienced through the cumulative opacity of these lenses. This is why any reaction to A Small Place includes anger, likely casual dismissal: one spasm of concern for human rights excuses hundreds of years of subjugation. We defend ourselves against the criticism. We point out that American dollars improve the economy. In our blissful escapism we tip more, smile more, share more of ourselves with these delightfully backward blacks. We notice this; we feel guilty, we might not. Either way, we lock our valuables and passports in safes provided in every room.

Antigua is representative of all Caribbean nations, as seen in the application of the opening narrative to a documentary on economic imperialism on Jamaica. Antigua is pan-colonial also in that it is indicative of the failure-to-launch of Caribbean nations. We might say: in the last fifty years why have you not improved your situation? Why have you not made progress, expelled your corrupt leaders? Why is your country so poor, so dependent on the patronage of wealthy foreigners? How is that you cannot climb out of the sulfurous pit of tropical hell we dug together? Why did the revolution on Haiti flounder, when its role models resulted in such exalted and liberal lands of hope and glory? True, an island does not have hundreds of thousands of diverse square miles of land to supply nearly every imaginable resource. Also true, Antigua has only had fifty years to develop toward self-sufficiency: something we have been perfecting for hundreds (and only through trade: not very self-sufficient). Meanwhile, Antigua and Jamaica sit along a thoroughfare of Atlantic trade like Estragon and Vladimir Waiting for Godot. They (the physical islands) occupy themselves, engage in the recurring and reversing drama of master and slave, but ultimately seem to be awaiting deliverance (fruitlessly).

As I write I pass a highway billboard advertising Jamaica. More correctly, an airline sign exclaiming "More Jamaica! More stops...". There is no picture. There is no mention of socioeconomic difficulty. There is no character exposition, just the invocation of the island as a destination. New Englanders will flock there soon, migrating to warmer climes. They will drink, with no thought that the rum that quiets private demons is an industry made successful in the triangle trade by the cane-slashed arms of slaves. The spring breaker will likely have the closest encounter with the shadows of empire. Some will be scared and excited by the sojourn into realer Jamaica to find drugs that so ubiquitously defines the country after the popularization of suburban Rastafarianism. Dollars in the illegal drug trade are of as much use to the local community as those buying daiquiris. The economic stimulus of tourism funnels into the manufacture of rum, the construction and renovation of hotels, the erection of beach-sequestering fences.

It is our imposed Capitalism and Democracy (big C big D like G-d) that keep them adrift. Easy money is much preferable to hard-earned pittances. So yes, every entrepreneur in the Caribbean could leave the tourism trade. Or close an international bank, casino, bar, dock, boxite mine, oil refinery. All the laborers in these semi-lucrative ventures (lucrative for investors) should go and, well... farm? Fish? There's no money in that, and anyway aren't we instructing them on how to regulate the fishing industry for sustainability? In the name of environmental stewardship we transfer our system of reserves and limits and licenses that further limit job availability. That’s not such a problem, since the countries are too poor to police their waters. The licensors are too poor (or greedy, tomato tomato) to not take bribes. Anything produced from the fertile volcanic soil is inherently more expensive than its counterpart grown on tracts of field as large as some of these island nations, and harvested by an unimaginable fleet of machines. Any mineral exhumed from the volcanic soil will have a pronounced environmental impact on these oases, and we will probably tell them to stop lest our vacationing beaches are near dead reefs instead of live ones. There is not the infrastructure to cheaply manufacture goods, and the nearness to western markets doesn't compensate for this in a world supplied by containers ships and jet liners. There is no god but Profit, and his prophet is Capitalism. In his service the Antilles have no impetus to harvest food by the bounty of the earth, given by that second-tier god. In his service there is no better option than corruption, need rationalizes crime. At this utilitarian (sensu John Stuart Mill) level the distribution of happiness (resources, etc.) has no real meaning, since there is much too little to go around. So maybe people get jobs elsewhere? We don't want them to take good American jobs from good Americans. We don't really want them to take bad jobs that most Americans don't want. But they are here and those jobs are available and maybe there's subconscious satisfaction for some that the those once forced into unsavory work now choose it.

We can be proud of successful Antiguans, because they are Antiguan-Americans. Their success is a product of American education, and American hope and prosperity. The money they send home is lost to the GDP, but we get their banks fees and international phone bills (assuming there is a phone to call to, maybe that is an early purchase with the remittances). We can be proud of Kincaid, because she is an American who champions international causes. We can discount her should we feel like it, because she is an American who champions international causes with the voice of Antiguan bemoaning local issues. Ex-patriotism is successful on our terms.

And we are back to where we were before: Kincaid, ex-patriotism, white condescension, exploitation ad nauseam. Any argument of Caribbean history and politics and culture is necessarily cyclical. Or tidally recurring? Or more likely, things just never changed. Haiti had a violent revolution, which could have been productive (in terms of culture building and nationalism; in terms of economy) had they not razed all their infrastructure. Most other island nations were discarded as no longer profitable to the white master. Imagine not claiming our own independence! Imagine being “forgotten, then remembered, then not important enough to go back for” (Kincaid 16)! Imagine (because we have the luxury of acknowledging that things could be worse) dependence on patronizing visitors, for whom the nicest buildings are built, and the smoothest roads paved, and the most pristine of nature preserved. Imagine being robbed of wallet, and cultural dignity, and the choicest real estate. Imagine owning a fishing boat and motoring in the shade of cruise boat because that’s where the large grouper gather (the reef is already overfished, or maybe dynamited for a shipping channel). Imagine all this, and say without doubt in your voice that Kincaid (or the narrator) is whiny, possibly deranged and quite stupid. She offers no solution, but what solution could there be for a problem so developed in a place so small and undeveloped.

Diatribe is a fitting literary device for the tribes of the African diaspora. Swiftian satire (A Modest Proposal) is a western phenomena, not well-suited to the humorless situation of Antigua: an insolated and isolated island struggling in the shadow of the West (of which it is, geographically though not socially, a part). A Small Place is a commentary, arguably a plea for cease fire. Antigua doesn't hate us for who we are, about which someone who has never left the island only knows through global commerce. A common reaction to Kincaid's attack is withdrawal, I am not who you think I am, I am who I think I am. How Cartesian. We think we are our truest in our homes, surrounded by people we actively try to please in our actions. We behave in places where we know the laws and consequences, and have duties to fulfill which require us to show respect and diligence and reserve. Americans in America certainly rude, but most often in contexts where there is no long term consequence or discomfort. English in England are tolerable too. But give a people a playground on jeweled shores where their trash and manners have no meaning beyond seven days hence and see what happens. Like animals, we avoid shitting where we eat. We are actually more ourselves, base selves, in this playground. The Antilles hate us because in “unwinding” we cast off human decency and any measure of selflessness to indulge the senses at a cost to them. We remove ourselves entirely from the small place theatrics of an island, and expect the lack of regard (good, bad, neutral) to go unnoticed. The people talk. Because they work in the tourist industry, they talk about the obnoxious avatars of America. Because of the smallness of the place, they talk much and to many people. Like resonating waves, the implications of our intrusion build and build. In a way A Small Place is humorous, or at least the polarized pageant of “discussion” surrounding it. It is humorous because the reactions are so violent (for every reaction there is an equal and opposite reaction). It is also humorous because those who take greatest exception to it are the contrarians most likely to escalate their own very small problems born of very small communities into catastrophe and gusto. Kincaid doesn't suggest a socioeconomic solution for Antigua because there isn't one which will work in the current system. She suggests, catalyzes, a thought catharsis. A reaction to A Small Place says much more about the reader's self-involvement and power of denial (conversely: their ability to stomach criticism) than the text says about the author.


I had a long, fascinating conversation yesterday about building momentum in the North Atlantic blue economy. To be clear, I do not think that person is an asshole. 

The key takeaway was that the "old" way of funding and governing Ocean innovation (whatever that is) doesn't serve the public as well as practitioners believe. Inclusivity and cooperation are the way forward. 

I agree.

But inviting more people to the table has inefficiencies, which most people are not prepared for or interested in dealing with. When advocates talk about one community of academic, industry, conservation, and government united in defense of the World Ocean, I think about how hard it is to make decisions in an organization of 10-30 people with existing social and political affinity.

I helped run all-volunteer community radio stations for a while, and there was a spectrum of contributors, with three types that stood out. "Neophytes" were enthusiastic newcomers that just wanted to help in any way possible. "Jockies" were seasoned entertainment or broadcast industry people that liked to work pro bono and just do their thing. "Managers" like to get-shit-done, and sometimes created boundaries to help do that. And there's the board, but we don't talk about the board.

People either get frustrated and move on, or they become "managers", in what is supposed to be the flattest of flat organizations. If you have ever been in the Academy, gov't, or a business while they try to be flat, I am sorry. Mostly because moving on is a bigger deal, and the sunk cost fallacy keeps a lot of people trapped. 

Bringing together orgs with parity might be possible, but they are still deeply hierarchal. So, a team of four orgs is mostly a team of four people, or at least was initiated by them. And we praise this, or at least reward Exclusivity. Because that's where ALL value comes from in our present economy. This might be changing, at least a lot of people are betting that it is. But there are still plenty of gatekeepers. 

According to a friend, the optimal number of gatekeepers with which to form a cabal is seven. Too many more and the knives come out, too few and someone might be held accountable. If you are a secretive cabal, your work will be easier, because you can just do crazy shit and no one ever has to know.

The oath to join a secret cabal is called a non-compete and non-disclosure agreement. Inter-cabal communications are called mutual NDAs. Mutual NDAs only exist because eventually everyone realizes that Not Invented Here is a Really Bad Idea, and decide to "collaborate". 

Not all ideas are good ideas, and even some good ideas have unintended consequences. So someone has to draw a line somewhere, but it shouldn't be about who gets to participate and make decisions. And if social entrepeneurship is going to play a part in making Earth a more hospitable Ocean planet, we can't be forming elite teams and exclusive clubs. 

This is why the public statements "we have to keep the yahoos out" from an Executive Director of an industry association trouble me. Or private statements about entitlement to the Ocean based on family history. Or landowners that don't want to look at aquaculture.

All of these people feel that they have earned executive power by tenure. I was here first. But the first is rarely the penultimate, right? We learn that first is best very early in life, but that's only true in ranked competition, aka sports stuff. If winning the race doesn't matter, you can just do Great Work.   

It's not enough to be working towards carbon neutrality or social justice, or even have achieved a personal nirvana. Especially in new ventures, because people always underestimate the investment and take shortcuts. Always. Always! Less than ten percent of startups succeed, and they must offset the damage done by the death spiral of the other ninety percent. See, the ninety percent said they'd get to Sustainability and Accessibility, like later or whatever. POST revenue. RIP what a waste.

When all you want to do is get in the Action, you justify things. All is fair and love and war. Big, hot topics like Climate Change and other Ocean Catastrophes tend to evoke combative and sentimental tendencies. 

We can draw another lesson from radio. Advertising on non-profit radio is not advertising, it is sponsorship, and there is a very specific rule: "no calls to action".

I like this because it forces you to use factual statements about a sponsor, and defers to the listener to Make A Decision. There is a kind of magic in someone taking the initiative themselves. There is no magic in telling someone what to think.

In terms of these big ideas and the Blue Economy, I would like to also institute the policy "no calls to arms". I don't really want to unpack that, except to say that climate change is not a thing that can be killed, and none of our wars-on-x did much except justify hideous behavior.

What people want is accountability, I think. Adversarial encounters between Ocean agents are going to increase. SOME folks are entitled shits, but most are curious and open to things that are comprehensible. If all the things in the R&D pipeline for the North Atlantic pan out, there will be a huge increase in the number of vessels, structures, businesses, robots, and humans on/in the water. 

All of them will be waiting for you to mess up and be held accountable. Do better. Communication takes courage.

By adopting observability as a policy, you prove you hold yourself accountable. To me that means you know (quantitatively) your effect on the world, you show that face publicly (and not aspirational marketing), and you plan for the worst. The rest will follow.

My gratitude goes out to everyone trying to make a difference.


## Purpose

Our public presence is primarily through digital media, so attention to presentation and consistency in these channels is important to us. To that end, this covers the types of media and brand marks we invest in. I hope it will reduce bike-shedding and agile revision spirals.

Explicit design policies are provided. These exist to reduce friction, revise when appropriate.

## Marks

Our company is Oceanicsdotio LLC. The short operating form, in order of preference, is one of:

- oceanics.io (non-proper noun, the web platform)
- Oceanics.io (proper noun, the company)
- Oceanicsdotio (sometimes “.” is inappropriate for context, the company)
- oceanicsdotio (the platform)
- oceanics-io (cases where “.” is not a valid character, the company)

Product marks:

- Bathysphere (database and back-end)
- Neritics (analytics environment)
- Dagan (neural network / AI persona)

Short form copy and tagline examples:

- Situational awareness for a changing ocean
- Ocean analytics as a service
- The trust layer of the blue economy
- The **other** marine autonomy

## Copy

Language should be as straight forward as possible, while retaining technical density. 

Basically, know what you are talking about, and don’t dumb it down.

Less is more, and I personally hate adverbs. Sans serif almost always, except for print. Large fonts, moderate white space.

Don't use these, because, unless...

| **Word, Phrase, or Element** | **Reason, if given** | **Exception**         |
| ---------------------------- | -------------------- | --------------------- |
| leverage                     |                      |                       |
| vast                         |                      |                       |
| myriad                       |                      |                       |
| dynamic                      |                      | dynamical, literal    |
| synergy                      |                      | synergistic           |
| constellation                |                      |                       |
| fight                        | no call-to-arms      |                       |
| force multiplier             | no call-to-arms      |                       |
| blue                         |                      | blue economy          |
| lighthouse                   | saturated in Maine   |                       |
| pine tree                    | saturated in Maine   |                       |
| sustainable                  |                      | literal               |
| with a purpose               | Sofar                |                       |
| power                        | no-call-to-arms      |                       |
| progress                     |                      |                       |
| premium                      | NOTAFLOF             | literal, $            |
| Maine                        | saturated            | stakeholder not brand |
| advanced                     | demeaning            |                       |
| AI                           |                      | funding requests      |
| nexus                        |                      |                       |
| green                        |                      |                       |
| basic                        | demeaning            | chemistry             |

## Visual media

No stock photography or “Corporate Memphis”.

Images should support written copy, and be licensed from photographers/artists when possible. Creative work sourced from independent designers is preferred. It is not acceptable to use predatory multi-artist bidding systems.

Video is a major web performance and accessibility risk.

Video should be used sparingly, for purposes to which it is best suited, especially conveying large amounts of information quickly in a educational or news context.


# Docker 

Docker is a container service for hosting web or local applications. Containers are use resources more efficiently than virtual machines. Deployment across many machines or multiple clouds needs to be orchestrated. We will use Kubernetes to run multiple Python Flask applications, and map these services to ports using Nginx as a reverse proxy.

## Install

Docker can be installed on natively on Linux or inside a virtual machine on macOS and Windows. It works on Raspberry Pi and NVIDIA Jetson development platforms. 

All commands assume you have root privilege. 
If something doesn't work, or you don't want to default to root, prefix commands with the usual `sudo`.

### Raspberry Pi

For the RPi, you can use a script provided by Docker,  

```bash
curl -sSL https://get.docker.com | sh
usermod -aG docker pi
```

### Jetson TX2

A TX2 running Ubuntu for Tegra (U4T) will need some utilities, so do `apt-get install curl nano`. 
You will have to flash the TX2 with at JetPack >= v3.3 for there to be container support. Install manually with, 

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install docker-ce
```

If you want to start the service immediately, 

```bash
systemctl enable docker
systemctl start docker
```

# Kubernetes

Container clusters can be used with Docker Swarm, but if you get serious about deploying microservices, you'll need to learn the industry standard Kubernetes (k8s). This is a [container orchestration and clustering](https://youtu.be/v77FFbQwC6E) tool which came out of Google. 

There are a boggling number of certified providers popping up. Depending on who you use as a cloud computing platform, your tool of choice might vary. 

We maintain three environments:

* Digital Ocean managed cluster for production
* single-node `minikube`  bundled with Docker-for-Mac for development
* Native installation with x86-64 controller and ARM64 workers 

RPi don't really cut it as controllers, and get bogged down quickly. Ubuntu 16.04 is (at time of writing) required for flashing the TX2, and works well as a controller on a Intel Mac. The TX2 can also run the control plane if you're not doing interactive work.


## Install

### Preparation

The bare-metal cluster is composed of System-on-a-Chip (SoC) nodes running Raspbian Lite or U4T Linux distros. Linux uses swap space to exchange data between memory and disk, but is not supported by k8s. First disable and remove it,

```bash
dphys-swapfile swapoff
dphys-swapfile uninstall
update-rc.d dphys-swapfile remove
```

Use the text editor of your choice (`nano` is easy, and we installed it earlier), to append `cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1` to the `/boot/cmdline.txt` file. Then `reboot`. 

> The `cgroups` (control groups) kernel was introduced to Linux by Google contributers, so no surprise that k8s requires it. The `cgroup_memory=1` option is probably unnecessary (it does the same thing as `cgroup_enable=memory`, and was introduced as a requirement temporarily during an update to the Raspbian OS).

### Setup

Get repository info and install the [admin interface](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) package,    

```bash
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update -q
apt-get install -qy kubeadm
```

You initialize a cluster on the managing instance with `kubeadm init`, which will provide next steps and instructions for connecting workers.

The cluster needs a container networking interface (CNI). Weave, Calico, and Flannel come up often. Weave can be applied with, 

```
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
```

Weave Net implements the [cluster networking model](https://kubernetes.io/docs/concepts/cluster-administration/networking/) required for k8s. Specifically, all containers within a pod share a single IP address and can access each other on `localhost`, and all containers/nodes can talk to all containers. 

As workers join the cluster, they will visible to `kubectl get nodes`.  

### Troubleshooting

Some libraries may be needed for communication, certificates, and encryption,

```bash
apt-get install apt-transport-https ca-certificates curl software-properties-common
```

You may also need to edit the repository list so apt-get can fetch dependencies (socat), `nano /etc/apt/sources.list`.

## Algorithm

A convex hull is a container around a group of points (a subset of those points), which has inward acute angles only. This can be any number of dimensions, but is frequently in two for spatial queries. Convex hulls are a subset of the space define by a bounding box (also called extent), and superset of the true polygon.

A simple [convex hull algorithm](https://www.oreilly.com/ideas/an-elegant-solution-to-the-convex-hull-problem) using some linear algebra is as follows:

1. Choose left- and right-most points <Equation text={"\\vec{U}, \\vec{V}"}/>
2. Include <Equation text={"\\vec{U}, \\vec{V}"}/> in the hull, and bisect the set with <Equation text={"\\vec{U} - \\vec{V}"}/>
3. For each half, choose the furthest point <Equation text={"\\vec{W}"}/>
4. Discard points inside triangle <Equation text={"\\Lambda (\\vec{U}, \\vec{V}, \\vec{W})"}/>
5. Repeat recursively with <Equation text={"\\vec{U} - \\vec{W}"}/> and <Equation text={"\\vec{V} - \\vec{W}"}/>

Array libraries with linear algebra, like `numpy`, make this simple. You partition the set by the sign of the cross product with the vector between starting points. We’ll pass in the array by reference, along with indices, and `convex_hull()` will return indices required to reconstruct the hull from the original. This way we don’t need to copy, change, or destroy large amounts of data.

The indices of each half are passed to `segment()` which determines a far point by the cross product of <Equation text={"\\vec{U} - \\vec{V}"}/> and <Equation text={"\\vec{W}"}/>. The sign of the cross products of <Equation text={"\\vec{U} - \\vec{W}"}/> and <Equation text={"\\vec{V} - \\vec{W}"}/> with <Equation text={"\\vec{W}"} /> are used to choose new members, and the process continues until there are no candidates remaining.

## Implementation

The following is a Python function that does this, and saves an image:

```python
from numpy import random, argmax, argmin, cross, argwhere, arange, array, hstack, vstack
from matplotlib import pyplot as plt


def segment(u, v, indices, points):

 if indices.shape[0] == 0:
 return array([], dtype=int)

 def crossProduct(i, j):
 return cross(points[indices, :] - points[i, :], points[j, :] - points[i, :])

 w = indices[argmin(crossProduct(u, v))]
 a = indices[argwhere(crossProduct(w, v) < 0).flatten()]
 b = indices[argwhere(crossProduct(u, w) < 0).flatten()]

 return hstack((segment(w, v, a, points), w, segment(u, w, b, points)))


def convex_hull(points):

 u = argmin(points[:, 0])
 v = argmax(points[:, 0])
 indices = arange(0, points.shape[0])
 parted = cross(points[indices, :] - points[u, :], points[v, :] - points[u, :]) < 0

 a = indices[argwhere(~parted)]
 b = indices[argwhere(parted)]

 return hstack((u, segment(v, u, a, points), v, segment(u, v, b, points), u))


groups = (
 random.random((100, 2)),
 0.5*random.random((100, 2)) + 1,
 0.5*random.random((100, 2)) - 1
)

hulls = tuple(map(convex_hull, groups))
hullsUnion = vstack(tuple(group[hi, :] for hi, group in zip(hulls, groups)))
union = convex_hull(hullsUnion)
pts = vstack(groups)
subset = convex_hull(pts)

fig, ax = plt.subplots(1, 2)
ax[0].set_title("Convex hull of all points")
ax[0].axis("equal")
ax[0].scatter(pts[:, 0], pts[:, 1], color="black")
ax[0].plot(pts[subset, 0], pts[subset, 1], color="black")

ax[1].set_title("Convex hull of hulls")
ax[1].axis("equal")
ax[1].plot(hullsUnion[union, 0], hullsUnion[union, 1], color="black")
for hull, group in zip(hulls, groups):
 ax[1].plot(group[hull, 0], group[hull, 1], color="black")

fig.tight_layout()
fig.savefig(fname="convex-hull.png", bgcolor="none")
```

This can be used as a library, to help handle a lot of polygon data. The convex hull of the union of hulls, is also the convex hull of all points contained. This means you write nice `map()` and `reduce()` functions to sort and associate complex shapes without needing to do point-in-polygon calculations.

<img 
 src="assets/convex-hull.png" 
 alt="Example of a convex hull"
 style={{
 filter: "invert(1)",
 }}
/>

However, only aggregation works this way, and not removing convex hulls. If a polygon is going to be removed, the union will need to be recalculated from the source data.

The code can also be deployed as a simple service, similar to thumbnail image endpoints for cloud storage. Putting this info in the header of an object store lets you identify and prioritize datasets of interest, without them being fully ingested into a database that supports and is set up for spatial queries.


This is the body of a public comment submitted to NOAA in response to their [draft artificial intelligence strategy](https://nrc.noaa.gov/Portals/0/Draft%20NOAA%20AI%20Strategy_v1-6.1_11.13.2019_update.pdf?ver=2019-11-14-111103-647)

It is difficult to convey the mixture of excitement and disappointment at reading the draft artificial intelligence strategy. Excitement in spirit, disappointment in body. Given widespread evidence of successful adoption, you determined AI low-risk, and of national importance. Your goals in the service of efficiency, effectiveness, and coordination are in command-and-control terminology, and implicitly structure execution and assessment of the strategy. I would like to provide this public comment, as an oceanographer and software engineer working on AI applications in service of prosperity, agency, and cooperation.

## Structure

> NOAA will create a Center for AI (NCEI) and an AI technical committee. These entities will prioritize related budget lines, develop an "open source" science gateway, adopt cloud-native computing, organize events, and have executive-level conversations. The outcome and performance indicator is "[exponential] increase in use and utility across all of NOAA".

I question the need for a permanent center. Administratively or geographically concentrating domain expertise works against boundary spanning, and excludes participation. The performance metric here manages to be both vague and over-optimistic, simultaneously consolidating authority, mandating action across all offices, and pushing those services back to the public. Considering there is a major current of concern over ethics in AI, the committee or other bodies should have as much transparency and inclusion as possible.

An open source gateway is great, but exposing the inner workings of the data and service access layers doesn't do anything for observability and interoperability of models and services. You end up with arbitrary data format requirements getting baked into RFAs (EPA, WaterML). The more industry functions and standards that NOAA assumes as canon, the less incentive there is for innovation and competition in industry.

## Innovate

> NOAA will innovate by institutionalizing AI and adopting a "requirements-based" process. This process will be continuously updated to keep pace with technology. Funding priorities will be communicated through FFO and RFPs, and prize competitions. Adoption will be tracked using the internal NRDD, and research projects at partner institutes evaluated with yet-to-be developed metrics. Promising methods will be evaluated through multiple "testbeds".

Innovation is not meticulous curation and project management, nor is it sourcing your next idea through predatory practices like competitions. 

Innovation is giving creative people resources to synthesize new ideas, and getting out of the way. It is not the job of the government to pick winners and losers. Your process will not keep pace with technology, and adding more steps in the development pipeline through performance monitoring and gate-keeping will undo gains from automation. Internal tracking in NRDD with annual public reports is not an improvement, and doesn't contribute to the open community of practice that has inspired this strategy.

The most innovative thing, would be to proactively seeking out those working in the space you propose to occupy, and enable constructive progress toward analytics products with net social and environmental gains. To date, NOAA has limited experience with artificial intelligence, exploring applications in robotic survey, data quality, and image analysis.
 
It is distressing that the applications cited are image-based analysis tasks using decades-old methods, while ripe areas of human-machine interaction, graph reasoning, risk analysis, and behavior are unacknowledged.

## Operationalize

> NOAA will "rapidly accelerate" from basic research to algorithmic products, by taking things that already exist in industry and academia, and developing internal technical documentation that will be updated once per year. They will then evaluate these adopted technologies for re-commercialization, if they pass the pre-operational test phase. Annual reports will be used to build awareness and evaluate investigator performance.

What is the jounce on that rapid acceleration, and how you are going to get there by only assessing the state of knowledge once per year? We are in the middle of unprecedented environmental crises, so I agree that speed it essential. Live a little, distribute risk. 

Fund many projects which seem likely to fail. Fund identical projects in different places. Get people talking and co-operating on issues, provide a non-prescriptive forum for that. State clearly your needs and how much you are willing to pay, and encourage federated solutions that enable small companies and labs to compete with vertically integrated contractors and service providers.

## Partner

> NOAA will expand partnerships in AI, with a focus on the academic/research communities, interagency cooperative agreements, and public-private agreements through CRADA and SBIR. They will contribute to NSTC committee on AI, and engage in conference and workshop circuit.

Using existing approaches will result in status quo results. To go faster we need new funding and partnership mechanisms that enable small high-risk research projects by diverse participants, and incentivizes the validation and replication of work that goes into training and parameterizing models. We need methods to crowd-source knowledge, and rapidly disseminate it. Be a facilitator, not a domain expert. Sending executives to conferences is not a national strategy.

## Train

> NOAA will provide their staff with tailored training, using existing programs, partner guidance, and MOOCs and such. Staff will be rotated through offices to enhance cross-pollination, and positions and performance plans will be updated. They will support and lead collaborative conference and workshops, and "actively encourage" grad and co-op training to improve recruitment.

Notably absent is a focus on training a general workforce, discussion of diversity or inclusion, much less training in how AI and ML can contribute to inequity. There also seems to be an emphasis on optimizing people, without much attention given to the effects of moving people or altering their positions. People are understandably uncertain of their professions.

The decentralized use of AI resulted in redundancy and limited scope, but those successes were through the agency of individuals. Data literacy is crucial in the near-term, but often the way people come into it is through problem -> solution logic. Encourage the practices and people that arose organically. Making it a supply problem before there is a demand is premature optimization. It's hard to get people because your recruiting pipeline sucks, not because there is a shortage of people who can learn on the job.




## Risk factors

Many factors contribute to accidents, but basically it boils down to "these things will kill you":

* ignoring warning signs and limits
* bad air management
* bad buoyancy control
* dangerous ascents

### Ascents

At depth gas is pushed into blood. Staying at depth and then ascending too fast causes gas bubble out into tissue (Henry’s Law).

Air in the lungs can expand and cause trauma can if holding breath while ascending (Boyle’s Law).

If you stay within your no decompression limit (NDL) and ascend no faster than 30fpm, you don't need a safety stop. This is almost always prefable. Bottom line, plan the dive within NDL, and limit depth exposure. You can follow a sloping bottom, or vertical reference line to help maintain a consistent speed.

When surpassing the NDL, you "must" stop at the ceiling, plus 3-5 minutes at 10-20 feet. It never hurst to take a safety stop at 20' just in case. In an emergency breath out continuosly and maintain a controlled ascent.

If you ascend too fast, slow down!

If you miss a safety stop, return to depth and wait it out if you feel ok!

### Air management

Pressure doubles every 10m, so more air is consumed. Use the rule of "Rule of Thirds". From initial PSI, use 1/3 each for descent/bottom time, return trip, and ascent/safety. Leave a 500 PSI reserve for emergencies.

Over exertion causes build up of CO2, or hypercapnia. Symptoms include shortness of breath, headaches, confusion. Avoid this by minimizing workload and staying horizontal with proper body trim. Breath deeply and slow down. Do not exceed 100’ and stay shallower if possible.

N narcosis causes drowsiness and mental impairment. It is apparent at 100’. Oxygen toxicity also occurs when using enriched air nitrox (32-36% O2)

Make sure you:

* Check equipment before dive for problems like leaking o-rings or freeflow regulator
* Check gauge at start of dive
* Estimate how long the supply will last
* Agree on tank pressure exit thresholds
* Be aware of current, and head into it to start
* Check gauge regularly during the dive
* Be conservative if the dive is strenuous

### Physical and mental preparation

Be in good health. Minimize anxiety by planning, declining to dive if necessary, and building confidence gradually. Entry level divers should only descend to 60’ in good conditions. Reduce risk by staying shallower while you continue to train, and choosning sites that match group skill level.

Assess personal health on the day of the dive. Cold symptoms in the ears and sinuses can take 7-10 days to resolve, so be honest with yourself and others.

Assess traffic, visability, temperature, entanglement risks, currents, and wave height before getting in the water.

If you are diving with new people or in a new place be sure to note:

* staff training
* professional attitude
* cleanliness and order
* equipment quality and maintenance
* air quality, tank inspections
* emergency oxygen availability
* boat safety, emergency action plan
* entry and exit procedures, swim line

### Buoyancy

A positively buoyant wetsuit requires weight, but at depth the suit is compressed and you will become negatively buoyant. The weight difference between full and empty tank is 3-4 lbs.

Use correct weighting, so that you float eye level at the surface with the BC partially inflated. Your center of gravity should be adjusted so that it is easier to maintain a horizontal position. If over-weighted, compensating with BCD will increase drag.

Remember, you are using air to change buoyancy. Switching depths wastes air. Don't do that.


## Background

Cyanobacteria are notorious for the dense, green scums they form on water bodies the world over. 

Blooms are increasingly common due to changing nutrient regimes and changing temperatures, and this means more incidents of human and animal toxicity. Because, of course they produce potent neurological toxins. 

The genera Anabaena and *Microcystis* share similar phenomenological behavior. They out-compete con-specifics by using specialized stages and cells for quiescence, acquiring nutrients, and creating buoyancy.

The buoyancy bit is the most interesting. Both types use a combination of lighter-than-water proteins and gas vesicles, but the method of controlling the effect varies. 

Anabaena relies on vesicle collapse, basically purging gas to descend like a diver. This results in a hysteresis, where the method of action is different for increasing or decreasing buoyancy. 

*Microcystis* instead chemically alters the ratio of carbohydrate and protein move, like, well, a chemical ballast.

People I have see embark on a dynamic buoyancy engineering project do the first bit, while thinking they are doing the second. The second is harder to achieve, but provides better control. I hope engineers new to marine operations will incorporate solid, displacement, and dynamic buoyancy, as this can be safer and prevent catastrophic loss. 

To incorporate any vertical moving agent into high-resolution hydrodynamic models, or to have a hope for remote autonomous control you need a rigorous definition of vertical movement. 

The *Microcystis* way relies on the balance of carbohydrate (C) and protein (M).

<Equation text={"{{\\delta \\vec{M}} \\over {\\delta t}} = \\vec{C} \\cdot (f_1(T) - f_2(T)) - \\vec{M} \\cdot f_3(T)"} />
 
<Equation text={"{{\\delta \\vec{C}} \\over {\\delta t}} = f_4(I) \\cdot \\vec{M} \\cdot (1-V_p) \\cdot ({{{C \\over M }_{max} - {\\vec{C} \\over \\vec{M}} \\over {C \\over M }_{max}}}) - \\vec{C} \\cdot f_1(T) - f_5(T) \\cdot \\vec{C} - f_6(T) \\cdot \\vec{M}"} />

The subscripted functions are physiological processes that depend on light and temperature. The details aren't really that important to the discussion, but you can get an idea of the data requirements from this system diagram,

<img src="/assets/neon-stella.png" alt="Rigorous Definition" style={{width: "100%"}}/>

In the end you want a density difference compared to the local water column, which is used to calculate the Stoke's velocity,

<Equation text={"V = {{2gR^2 \\cdot \\delta \\rho} \\over {9 \\eta}}"}/>

That's g for gravity, R for radius, rho for density, and eta for viscosity. 

The question that surfaces for shallow lakes and estuaries, is this: does the constrained depth and mixing allow for surface accumulation of biomass and toxins? Aquaculture poses the opposite challenge. That is, how to control position in the water column to prevent local build-up. 


For a case with uniform colony size starting at the bottom of a in a 1-dimensional domain, several things happen:

- High temperatures result in surface accumulation
- Toxin intensifies in the upper water column
- Colony size appears to modulate timing and depth of microlayer, but it's an artifact of "bouncing" in the movement integration algorithm

For deep water, particles stabilize at depth thanks to feedback loops in the photosynthesis driven cycle,

<img src="/assets/cyanobacteria-buoyancy.png" alt="Buoyancy Driven Oscillation" style={{width: "100%"}}/>

For shallow water, there's no time to generate carbohydrate sufficient for ballast, and metastable surface layers form based on thermal and light limitation,

<img src="/assets/cyano-vertical-layer.png" alt="Cyanobacteria Vertical Layer Formation" style={{width: "100%"}}/>


What can we expect from the future? Will it be a shattered landscape of ruins or a verdant utopia? One major factor will be pollution. I believe that in 75-100 years our world (mainly our atmosphere) is going to be quite on its way to ruin unless we make more of an attempt to stop atmosphere-deteriorating chemical from being released. We will need to find a replacement for such harmful products as aerosols, or we will be living underwater (the greenhouse effect will melt the ice caps and the sea level will rise). Another horrible thing will happen if we don't stop the poisoning of our world: the rare animals and even more common animals and plants will start to die off. This will disrupt the food chain, leaving only humans who will have to survive on synthetic foods made from algae, which will be the only thing thriving. Yes, that's what going to happen if we don't shape up.

The above is almost inevitable since not that many people really want to put forward the effort needed and so we must put more of this nations' and every other nation's budget into the space program. This way, when the Earth gets to the point when it not suited to live on, we can escape on space ships capable of taking us to a Lunar or Martian base where the human race can expand further from. One day we will have to escape and we might as well develop the capability now.


## General Biology

1. 
<Inline parenthesis={true} authors={["Audet D", "Miron G", "Moriyasu M"]} published={2008} title={"Biological characteristics of a newly established green crab (Carcinus maenas) population in the Southern Gulf of St. Lawrence, Canada"}/> looked at green crab biological characteristics at the northern limit of the crab distribution in the Atlantic. They were able to identify mating time, male-female population ratios, size, mating behavior in the northern regions, which contrast to about a month later than what is usually observed in Maine.

2. The population dynamics of green crabs were studied in an estuary in Portugal 
<Inline authors={["Baeta A", "Cabral HN", "Neto JM", "Marques JC", "Pardal MÂ"]} published={2005} title={"Biology, population dynamics and secondary production of the green crab Carcinus maenas (L.) in a temperate estuary"}/>. Carapace color throughout the year, proportion of crabs in molt, sex-ratio, overall growth production, and presence of larvae were characterized. This study could be useful in thinking about population dynamics in Gulf of Maine estuaries.

3. 
<Inline parenthesis={true} authors={["Berrill M"]} published={1982} title={"The life cycle of the green crab Carcinus maenas at the northern end of its range"}/> investigated the different stages of the green crab life cycle along the central coast of Maine. They found that crab matures at 2-3 years old, breeds 2-3 times, and has a generation time of 3 years. Compared to warmer water populations, these populations grow slower, mature later, and have a longer life span.

4. 
<Inline parenthesis={true} authors={["Crothers JH"]} published={1968} title={"The biology of the shore crab Carcinus maenas (L.) 2. The life of the adult crab"}/>, though much older, gives a holistic description of the behavior and natural history of the green crab, and could be useful for filling in general background.


## Reproduction, Development, & Molting

1. 
<Inline parenthesis={true} authors={["deRivera CE", "Hitchcock NG", "Teck SJ", "Steves BP", "Hines AH", "Ruiz GM"]} published={2006} title={"Larval development rate predicts range expansion of an introduced crab"} /> determined the survival and development of green crab larvae in non-native populations, at different temperatures. They conclude that the rate of development and survival indicate that the green crab may continue northward into colder waters, regardless of interannual temperature fluctuations.

2. 
<Inline parenthesis={true} authors={["Chang E", "Mykles D"]} published={2011} title={"Regulation of crustacean molting: A review and our persepectives"}/> discusses early endocrinological that showed the glands and hormones responsible. Additionally, describes recent experiments that have provided information on the cellular and moleuclar aspects of molting. Very involved, but could be a useful resource for deeper understanding of molting.

3. Probably more molecular than necessary, but 
<Inline parenthesis={true} authors={["Chang E", "O’Connor J"]} published={1979} title={"Arthropod Molting Hormones"}/> could be a good resource if someone wanted a more in depth overview of the science of molting, as it existed in 1979.

4. 
<Inline parenthesis={true} authors={["League-Pike P", "Shulman M"]} published={2009} title={"Intraguild Predators: Behavioral Changes and Mortality of the Green Crab (Carcinus maenas) during Interactions with the American Lobster (Homarus americanus) and Jonah Crab (Cancer borealis)"}/>conducted a lab experiment to inestigate the effects of the lobster on mortality and behavior of green crabs. The presence of the American lobster had dramatic effects on green crab behavior, leading to increased hiding and climbing, and reduced standing in the open. These effects are likely an indicator of the scarcity of green crabs in subtidal communities, and suggest a likely decrease in green crab predation of small invertebrates that are common prey for decapods in the ecosystem.

5. 
<Inline parenthesis={true} authors={["Lyons LJ", "O’Riordan RM", "Cross TF", "Culloty SC"]} published={2012} title={"Reproductive biology of the shore crab Carcinus maenas (Decapoda, Portunidae): a macroscopic and histological view"}/> is a histological study of the reproductive cycle of male and female green crabs was performed on the Southwest coast of Ireland. They found a strong male bias, and a system was devised to describe and stage gamete development. Annual female ovarian development was characterized relative to season, as well as potential copulation seasons for males. Developmental stages of oogenesis and spermatogenesis were described, which ideally might benefit management.

6. 
<Inline parenthesis={true} authors={["Poirier LA", "Mohan J", "Speare R", "Davidson J", "Quijón PA", "St-Hilaire S"]} published={2016} title={"Moulting synchrony in green crabs (Carcinus maenas) from Prince Edward Island, Canada"}/> studied seven groups of crabs for molting characteristics, and it was found that a synchronized ‘molting window’ occurs during July for male crabs. In this month, molting rate increased over the first half of the month, following a 5˚C increase in water temperature. The presence of a halo on the episternites of the carapace indicated the crab would soon molt. This is a first step in assessing a soft-shell, green crab industry.

## Dispersal, Impacts, Restoration & Mitigation

1. 
<Inline parenthesis={true} authors={["Bryan E", "Beal B"]} published={2015} title={"Interactions between the invasive Eruopean green crab, Carcinus maenas (L), and juveniles of the soft-shell clam, Mya arenaria L., in eastern Maine, USA"}/> worked to understand the predator-prey interactions between green crabs and soft-shell clams using deterrents. They found that green crabs consumed clams protected by predator deterrent netting, and could even do so without leaving signs of chipping or crushing the bivalves. However, some trials show that predator deterrent options worked compared to open controls, did not work between rigid and flexible netting, and did not work by raising netting above the sediment surface. More work needs to be done.
 
2. 
<Inline parenthesis={true} authors={["Cosham J", "Beazley K", "McCarthy C"]} published={2016} title={"Environmental factors influencing local distributions of European green crab (Carcinus maenas) for modeling and management applications"}/> investigated the most relevant factors shaping local, fine-scale distribution of the green crab, to compare how adult and juvenile life stages change with depth, biotic interactions, vegetation, presence of shelter, and salinity. They conclude that using a single model to anticipate distribution is inadequate, and that more complex and localized methodologies are necessary.

3. 
<Inline parenthesis={true} authors={["Duncombe L"]} published={2014} title={"Evaluating trapping as a method to control the European green crab, Carcinus maenas, population at Pipestem Inlet, British Columbia, Canada"}/> describes a trap in place between 2010 and 2012 in British Columbia has shown a female bias, and that efforts for green crab removal have changed the population structure. Duncombe did not determine whether there were affects on population size, however.

4. Though long and complex, 
<Inline parenthesis={true} authors={["Gehrels HB"]} published={2016} title={"A study of green crab (carcinus maenas) interactions, cannibalism, and a first approach to model the effects of harvesting on its populations"}/> worked on identifying the current status of mitigation efforts for green crab populations, and found that current removal programs have little to no effect on the population growth rate. Therefore, current methods of harvesting alone are unlikely to result in a reduction in annual abundance.

5. 
<Inline parenthesis={true} authors={["Grosholz E"]} published={1996} title={"Predicting the impact of introduced marine species: Lessons from the multiple invasions of the European green crab Carcinus maenas"}/> compared the ecological characteristics of three spatially independent invasions of the green crab to determine similarities across invasions. They found that diet preference and ecological impact were similar across the three invasions. By studying the degree of similarity, a valuable measure of predictability can be employed for future invasions.

6.
<Inline parenthesis={true} authors={["Grosholz E", "Ruiz G"]} published={2002} title={"Management plan for the European green crab"}/> is a management plan put forth to evaluate the feasibility of current mitigation efforts in the United States, as well as an outline for plans to coordinate activities among scientists. Large and involved, but this management plan seems like a current and practical resource potentially of use.



## Linked lists

Linked lists have directed graph structures consisting of nodes containing a value and a reference to the next node. The final node has a `null` reference that signals the end of the structure. Implementation requires a `head` method or value that points to the start of the linked list, and a `tail` that points to the last member. If the list is empty, these are null. 

Linked lists can extend without resizing. Large and constantly changing data will make better use of linked lists compared to arrays, which fill or require costly extensions. Inserting and removing nodes does not require shifting or leaving empty spaces. Links may be stored separately from data, if they are kept the same size and order, and indices are used to point into the data. Indexing is O(n), insertion/deletion at an end is O(1) if the element is known, insertion/deletion in the middle is search + O(1), and wasted space is O(n). 

Linked lists are used to implement stacks and queues, and as entries in hash tables to store multiple values for each key.  For large number and polynomial arithmetic, the list stores tuples of coefficients and exponents that break the values up in memory. They can represent sparse matrices so that zero-value data are not stored. Values are referenced in two linked lists, for the rows and the column, and each node points to the next value in both i and j directions. 

The trade-offs are sequential access, and greater memory use than contiguous arrays. Variants try to overcome these limitations, or improve the strengths further.

Linked lists are often used invisibly under the hood. We kind of use them for analytics pipelines. 

### Variants

- Doubly-linked: implements bi-directional traversal, so each node usually references two neighbors. XOR-linking can reduce this to a single reference. Some operating systems use double linking to track running processes/threads. Malware like rootkits may hide by operating outside these housekeeping structures. They are also good for previous/next features in software products like image viewers, web browsers, and music apps. 

- Multiply-linked: like doubly-linked, these organize data in a different way, such as sorting by properties. 

- Circular: tail connects to head. They may have just a tail, from which the head is inferred. These are good for cycling through processes in a computer, or implementing a Fibonacci Heap. 

- Unrolled: store multiple elements per node to improve caching, and reduce wasted space from pointers. These are similar to B-trees. 

- Self-organizing: frequently accessed nodes moved closer to the head for faster access. Compilers and LFU caches use this optimization. Nodes are promoted by swapping accessed nodes with a neighbor. Nodes may move-to-front (MTF) on every access, or track access, and are re-ordered when a condition is met. 


## Hash table

Hash tables (often called hash maps, or dictionaries) use hashing functions to map an index value to a bucket of values. On average these are faster than trees, which is why this is a common data structure in language standard libraries. If all values are known in advance, you can optimize hashing function.

As long as the hash function generates a uniform distribution search, insert, and delete operations are O(1), with O(n) worst case. The space complexity is O(n).

A linked list is often used as the entry, and may have the head stored in the bucket to reduce pointer follows. Above load factor (keys/slots) 10 use balanced search trees for accessing data.

We use hash tables extensively in attaching data to a mesh and grid structures. In these cases the index is a tuple of indices, and these access information about neighbor relations and memoized intermediate calculations.


### Variants

- Separate chaining: keys point to a list of values, usually with 0-1 entries. This works best when the bucket array can be pre-allocated to prevent resizing operations. 

- Cuckoo hashing: constant worst case lookup time

- Self-balancing search tree: reduce worst case to O(logn), used by Java8 HashMap


## Other stuff

There are so, so many other options and combinations. Useful types for distributed data systems include: [distributed hash tables](https://blog.keep.network/distributed-hash-tables-49721094403d), prefix hash trees, [minimum perfect hash functions](http://iswsa.acm.org/mphf/index.html), and LRU caches. 



![Welcome to Maine](assets/hello-world.gif)


## Introduction

Larval American lobsters (*Homarus americanus*) live suspended in surface waters, carried long distances by ocean currents. After their pelagic phase, lobsters become competent forward-swimming post-larvae (stage IV) and settle as they encounter suitable habitat (Factor 1995). Settlement is influenced by physical processes, but there is also evidence of complex navigation (Kingsford et al 2002, Cooper & Uzmann 1971). They use photo- and geotaxis in search of shelter (Kingsford et al 2002), can swim upstream, and even follow odors over long distances (Boudreau et al 1993). Foraging juveniles return to the same shelter night after night without visual cues (Karnofsky et al 1989). Adults embark on migrations >200 km, and displacement trials show that lobsters can home to known geographic areas (Cooper & Uzmann 1971). The similar migratory behavior of the spiny lobster (*Panulirus argus*) is accomplished by magnetic orientation, demonstrated by displacement (Boles & Lohmann 2003) and induced directional preference (Lohmann et al 1995) trials. Marine and terrestrial organisms are known to sense field inclination and polarity, and use geomagnetic cues to migrate (Cain et al 2005). The navigation capability of *H. americanus* anecdotally suggests they might share this sensory mechanism. This study demonstrates re-orientation in response to manipulation of the local geomagnetic field. 

## Methods

Lohmann et al (1995) used electromagnetic fields (EMF) to study navigation by *P. argus*. Two square coils in series create overlapping EMFs with an area of magnetic uniformity (Firester 1966). By applying appropriate voltage, the apparent geomagnetic polarity within this area can be reversed. For this design, 22 turns of 16-gauge insulated copper wire were wound around a PVC frame. Side length $S$ was 150 cm, with a coil separation of 82 cm (Figure 1). The uniform EMF of this configuration has <0.5% deviation within a 50 cm diameter area (Firester 1966). 

The effect of rounded instead of square corners was ignored (pipe diameter = 5.08 cm). Firester's (1966) equation for the central field strength of a Helmholtz coil ($B$, gauss) was used with Ohm’s law to calculate the voltage $V$ necessary to reverse the magnetic field, $B=6.218 V S^{-2} R^{-1}$. 

The resistance of the conductor $R$ (Ω/ft) is important here. Flux density and magnetic field strength can be used interchangeably in this case (Diament 2000). Additionally, air and water have similar magnetic permeability, so $B$ is accurate in either media (Lohmann et al 1995). Magnetic deviation as great as 40° existed in some areas of the lab (attributable to HVAC). The coils were aligned using a compass in an area with minimal interference, such that north varied 5° between sides. 

The horizontal geomagnetic component for Boston has a magnitude of 0.198 G (National Geophysical Data Center), but *in situ* magnetometer readings were 0.238–0.245 G. Therefore $B$=0.478 G. Two DC power supplies (3.41 and 3.50 V) were connected to the coils. Coil resistance was 1.85 and 2.05 Ω, including power supply leads (Fluke 8050A multimeter). The difference in current was under <5 mA. The coils produced B=0.415 G, falling short of 0.478 G.

![Homemade electromagnetic coil](assets/helmholtz-coil.JPG)

Ten *H. americanus* from the New England Aquarium were used in baseline tests, and a subset of those in EMF tests. During the experiment they were stage VI-VII (CL=8.8±0.5 mm), and molted before trials began. Lobsters were kept in individual mesh tubes in a flow-through tray with aerated 15 °C artificial seawater, under a 12:12 hr light schedule. They were fed *Artemia* daily. 

Lohmann et al. (1995) covered the eyestalks of adult *P argus* in magnetic experiments to prevent orienting to the sun (Deutschlander et al 1999). In this study lobsters were blindfolded with black plastic, wrapped under the rostrum and eyes and secured to the carapace with SuperGlue. Blindfolded lobsters did not react to motion, and their ability to eat and walk was unaffected. A cotton thread was glued to the plastic on the dorsal carapace to limit movement to a radius short of the tank walls. Individuals made unsuccessful attempts to remove tethers immediately after affixing, but otherwise ignored them.

![Juvenile lobster with blindfold and tether](assets/lobster-leash.JPG)

The arena was a circular basin (28 cm ø). The bottom was covered with a uniform layer of mixed fine and coarse sand. The tank was centered in the coils at 75 cm height. Water was replaced between trials (depth = 5 cm) and sediment allowed to settle. A transparent plastic cover was divided into 30° sectors, with north marked, and a hole for securing the tether. A tripod-mounted DV camera filmed downward through the cover. The PVC frame for the coils was draped with black plastic sheet to prevent glare in the recording, and to reduce shadows that might bias unblinded trials. 

![Arena for behavioral experiments](arena.JPG)

In baseline trials (I) lobsters were corralled (4 cm ø) during a 5 minute acclimation, then freely wandered for 20 minutes. Video was converted to an image sequence with one minute intervals, starting when the corral was removed. EMF control trials (II) were conducted with blindfolds and tethers. There was no acclimation period. Runtime and photo interval were again 20 min and 60 s. For two trials the coil was activated at 10 min remained on for 10 min (IIIa). Two further trials had coil state toggled every 2 min (IIIb). One trial had coil state toggled preferentially while the lobster was traversing the arena, every 15–60 seconds (IIIc).

A reorientation event was defined as a 90° change in heading, not caused by interaction with the arena or tether. These corresponded to EMF reversal event if it occurred within 5 seconds. Continuous circling within a quadrant was evaluated as an event for every revolution. Events occurring within 60 seconds following any disturbance were ignored. Plots of vector averages were used to evaluate directional preference (Lohmann 1995). The number of distinct reorientation events (total and corresponding to reversals) was compared to runtime (considered in 5 second bins). 

## Results

Baseline trials (I) did not show obvious directional preference (Figure 2). Lobsters tended to reach the wall of the tank and follow it, reversing direction occasionally and at random. Most settled in the relative safety of depressions against the tank wall. This necessitated the use of a tether in control trials. Control trials (II) showed erratic movement (Figure 3), but no settling. 

Lobsters reaching the end of a tether would either (a) strain against it, (b) turn tangential to the radius and walk at the limit of the string, or (c) be lifted off the substrate by tension. Entanglement occurred in only one case, though in several cases the tether fouled on itself. This did not affect mobility, only the radius of movement. Slack was added in cases where the small radius caused constant directional change. If the lobster reached the wall, the tether was shortened, the next time it was not under tension.

In two trials (IIIa) a reversed horizontal field caused an instantaneous heading change (100%, N=2). One lobster was at the limit of the tether in the southern arena, heading east. It turned inward and completed a narrow loop before again reaching the end of the tether an resuming its counterclockwise course. The second was straining against the tether and moving clockwise in the northern arena; it also turned in and passed through the center of the arena, and reaching the opposite wall began a counterclockwise course. These behaviors were observed in (II). 

A series of reversals (IIIb) showed discernible reactions in some cases (44%, N=18), but the circular course imposed by the tether made it difficult to establish the cause of directional changes. This increased to 72% if including reactions that may have been related but were not the defined >90° turn. Of 27 non-random reversals (IIIc) performed over 11.75 min, 9 were ignored because the lobster was at the extent of the tether (including positive, negative and possible reactions). An obvious change in behavior occurred in 12 cases (67%, N=18). With a more lenient criteria for a related event there potentially would be 89% coincidence. There were 47 total reorientation events, occurring in 33% of 5 s time intervals. 

![Distribution of heading](lobster-weighted-vector.png)

Figure 2: average heading during baseline trials (I) (N=6). Direction is vector average of 20 positions, length is an indicator of course consistency. Following the tank wall results in a short vector.

![Distribution of angular position](lobster-circle-diagram.png)

Figure 3: Sector positions of two lobsters during blind tether trials with no reversals (II). This shows a possible bias to 270° (West), but seems coincidental when examining video instead of still images.



## Discussion

A major concern in evaluating frequency of reactions was the functional definition of what constituted a response. In general, non-associated events were given more leniency, and associated events only counted if there appeared to be no other justifiable cause. Simply reducing the correspondence interval after a switch in polarity to 3 s reduces the probability of a single event occurring in any given time bin to 29% and reduces definite responses (IIIc) to 50% of reversals.

If using instantaneous reactions (1 s), the probably of an event happening in any second is 0.07; and a definite responses occurred in 39% of reversals. Simplifying the lobsters behavior by giving more room to establish linear paths would aid in recognizing true events (and not throwing away potential data because of confounding factors).

Regardless, reorientation occurred in both moving and stationary lobsters at the instant of reversal. Particularly spectacular changes in orientation matched exactly the rotation of the compass arm used to confirm field change. Two other behavioral responses were observed. In cases of frequent reversal for more than 15 min two lobster made jerky vibratory movements and appeared agitated. This happened after reversals, and was followed by the lobster grasping one claw with the other, like wringing hands. When lobsters were stationary during reorientation, they would sometimes roll their tail in and pause before or during a turn. Lobsters would also stop and exhibit this behavior during field reversals. This might indicate that tail is acting as a sort of biological Hall effect sensor. Analyses of trace metals and electric fields in the lobster body might reveal more regarding the sensory mechanism. I have no explanation for the claw-wringing behavior. 

Because of the simplicity of component vector addition, three such arrays can be aligned to x- y- and z-axes, and allow greater control of the uniform area. If the design includes Hall probes to sense each axial component of the field, a CPU can dynamically shift the power supplied to the coils. A moored system in constant motion could maintain a field of uniform magnitude a direction regardless of its instantaneous orientation; DC supplied devices are also inherently suited to battery operation. 

The ability to change the apparent orientation of the Earth's field has applications in study of competent dispersal and population connectivity which may be driven by magnetic navigation. One possible application is *in situ* testing of larval orientation, particularly whether *H. americanus* post-larvae utilize a polarity compass in their settlement strategy and how this might affect species distribution. 

## Acknowledgments

Many thanks to the people whose enthusiastic generosity made this possible: Anita Metzler and the New England Aquarium for the lobsters; Eric Hazen and Paul Bohn of the BU Electronics Design Facility for power, parts and knowledge; Michael Ruane and Robert Kotiuga of BU Electrical and Computer Engineering for testing equipment and EM theory consultation; Jonathan Perry and Justin Scace for use of their tools and time; and Jelle Atema and Julia Spät for encouraging something ambitious. 

## References

1. Boles LC, KJ Lohmann (2003). True navigation and magnetic maps in spiny lobsters. Nature 421:60

2. Boudreau B, E Bourget, Y Simard (1993). Behavioural responses of competent lobster postlarvae to odor plumes. Marine Biology 117(1): 63-69.

3. Cain SD, LC Boles, JH Wang, KJ Lohmann (2005). Magnetic orientation and navigation in marine turtles, lobsters, and molluscs: concepts and conundrums. Integr. Comp. Biol. 45: 539-546.

4. Caprari RS (1995). Optimal current loop systems for producing uniform magnetic fields. Meas. Sci. Technol. 6: 593-597.

5. Cooper RA, JR Uzmann (1971). Migrations and growth of deep-sea lobsters, Homarus americanus. Science 171(3968): 288-290.

6. Deutschlander ME, JB Phillips, SC Borland (1999). The case for light-dependent magnetic orientation in animals. The Journal of Experimental Biology 202: 891-908.

7. Diament P (2000). Dynamic Electromagnetics. Prentice-Hall: Upper Saddle River, NJ. 497 pp.

8. Factor JR, ed (1995). Biology of the lobster Homarus americanus. Academic Press: San Diego, CA. 528 pp.

9. Firester AH (1966). Design of square Helmholtz coil systems. Review of Scientific Instruments 37: 1264-1265

10. Garrett MW (1967). Thick cylindrical coil systems for strong magnetic fields with field gradient homogeneities of the 6th to 20th order. Journal of Applied Physics 38(6): 2563-2586.

11. Karnofsky EB, J Atema, RH Elgin (1989). Field observations of social behavior, shelter use, and foraging in the lobster, Homarus americanus. Biol. Bull. 176: 239-246.

12. Kingsford MJ, JM Leis, A Shanks, KC Lindeman, SG Morgan, J Pineda (2002). Sensory environments, larval abilities and local self-recruitment. Bulletin of Marine Science 70(1) Suppl.: 309-340.

13. Lohmann KJ, ND Pentcheff, GA Nevitt, GD Stetten, RK Zimmer-Faust, HE Jarrard, LC Boles (1995). Magnetic orientation of spiny lobsters in the ocean: experiments with undersea coil systems. The Journal of Experimental Biology 198: 2041-2048.

14. Merritt R, C Purcell, G Stroink (1983). Uniform magnetic field produced by three, four, and five square coils. Rev. Sci. Instrum. 54(7): 879-882.

15. National Geophysical Data Center, NOAA. Geomagnetic Online Calculator.



Python is a dynamically typed, interpreted scripting language. Out of the box, this can be slower, but that is easy to overcome. You can do more with less code, programming in high level languages is faster.

Since the installation of the interpreter handles machine and operating system dependencies and quirks, you can write code once that will run anywhere.

The computational bits can also be refactored into lower level languages like Fortran and C.

## Just-in-time compilation

Numba does just-in-time (JIT) compiling on Python code to make it as fast as native C in many cases. This approach is good for pieces of code that run repetitively. 

## Parallel computing

There a several ways to accelerate code. This accomplished through parallelization in shared or distributed memory setting. Graphical processing units (GPUs) can accelerate jobs further. At the application layer, methods are encapsulated so that they can be executed as a batch of jobs across the processors on one or more machines

### Message passing

Python uses processor parallelism instead of thread parallelism. When processes need to communicate, they use Message Passing Interface (MPI).

Use the common C Pi example to see if MPI is installed:

```bash
mpicc call-procs.c -o call-procs -lm
mpicc C_PI.c -o C_PI -lm
mpicc MPI_08_b.c -o MPI_08_b.c -lm
mpiexec -n 1 call-procs
```

This will need to be working before Python bindings to MPI will work.

## Distributed computing

When parallelism expands outside of a single machine or hsared-memory space, we start calling it distributed computing.

For Python, [this](https://eli.thegreenplace.net/2012/01/24/distributed-computing-in-python-with-multiprocessing/) is a better description than I can come up with off the cuff. There are [two approaches to distributed computing](https://blog.computes.com/distributed-computed-centralized-vs-decentralized-c1d21202bde8): centralized and decentralized. Centralized is simpler, and will be discussed here. For ecological simulations, there will generally be some kind of controller that schedules the various tasks needed during each time step.

### Example pattern

Say you are calculating particle trajectories, and have a NetCDF output from FVCOM, but it doesn't include dispersal terms for whatever reason. 
The initial process is a job server/manager, which distributes work across machines and processes. 
The manager would do mesh partitioning, and hand off topology to the other machines. 
Time loops are handled by the server processes, which use asynchronous communication. 
Each job is for a single node in a mesh. 
The processes take initialization info, then wait for a forcing message. 
They take a integration step for each message, and report back the state variables.

For each time step, the machines would request a JSON of physical fields from the manager, and load it into memory. 
Then divide node/volume calculations among available processes. 
Once all the field calculations are complete, then you'd setup a particle queue, and have multiple processors do interpolation and advection until all particles are in the update queue. 
A particle leaving the local domain would go to a transfer queue, to be sent to another machine. 
Basically, the serial program needs to be broken down into a process manager, and self-contained kernels that have limited dependencies.

For program "Happiest particle", if we assume that a particle can never jump onto land, the procedures might be something like,

```
def A(array):
	"""Make some calculations across all nodes"""
	result = fcn(array)
	return result

def B(position, topology):
	"""Get interpolated value at position"""
	result = []
	for node in topology:
		result[node] = fcn(array, topology)
	return result

def C(positions, velocities, stage):
	"""Runge-kutta integration"""
	positions += velocities * fcn(stage)
	
def D(array): 
	"""find maximum"""
```

The psuedocode for the simulation loop would look something like,

```
Setup()
While(True):
    Map A(A) > processes
    collect results
    
    Map B > machines > processes
    collect results
    
    for step in integration:
    	Map C > machines > processes
    	collect results
    	transfer particles
    	
    	Map B > machines > processes
    	collect results
    
    Map D > processes > reduce > alert
    if (condition): 
    	print("All particles are ecstatic!")
    	break
    	
Output()
Exit()
```

What you really want is a [Docker swarm](https://docs.docker.com/engine/swarm/) to manage all that for you. 
Or more likely, Kubernetes. The idea being there is one "server" per processor, which acts as an addressable virtual machine. 
This does away with machine/processor distinction.


## Compute shaders

The GPU can act as an additional worker. Programming for the GPU is accomplished with compute shaders, which replace the traditional rendering pipeline. 

There on instructions online for [optimizing the Videocore GPU](https://petewarden.com/2014/08/07/how-to-optimize-raspberry-pi-code-using-its-gpu/) in Raspberry Pi, but it's probably not worth the effort.

[Arrayfire](https://github.com/arrayfire/arrayfire-python/wiki) is an open source optimizer for CUDA and [OpenCL](https://www.khronos.org/opencl/) on GPUs and CPUs, that has extensions for Fortran and Python. On MacOS, there is an installer package. After this has been dowloaded and executed, the Python package can be installed with:

```bash
pip install arrayfire line_profiler memory_profiler objgraph numba
```

The basic array creation functions and types are in `arrayfire.data`.



## Components

Networking commodity hardware to achieve higher performance is called a Beowulf cluster. **These instructions were created in 2017, and are out of date**.

The components available during training events includes single-board and system-on-a-chip machines.

| Description          | Vendor     | #    | Price | Address           |
| -------------------- | ---------- | ---- | ----- | ----------------- |
| Jetson TX2           | NVIDIA     | 1    | 299   | 192.168.001.105   |
| Raspberry Pi 3B      | Raspbian   | 4    | 60    | 192.168.001.100-4 |
| USB power, 10-port   | Sabrent    | 1    | 29    | n/a               |
| Micro USB cables, 3' | Sabrent    | 6    | 2     | n/a               |
| WL-520GU router      | ASUS       | 4    | 30    | 192.168.001.001   |
| Gigabit switch       | Netgear    | 1    | 50    | 192.168.001.138   |
| CAT5 cable           | Amazon     | 10   | 2     | n/a               |
| HDMI monitor, 7"     | SunFounder | 1    | 60    | n/a               |
| HDMI cable, 15'      | Amazon     | 1    | 10    | n/a               |
| Wireless keyboard    | Rii        | 1    | 28    | n/a               |

The Jetson TX2 has multiple CPUs, and a pretty serious GPU for accelerating jobs and video rendering. Raspberry Pi make [passable servers](https://www.copahost.com/blog/is-it-possible-to-run-a-web-server-in-a-raspberry-pi-3-as-a-dedicated-server/), even for [websites](https://www.readwrite.com/2014/06/27/raspberry-pi-web-server-website-hosting/).

## Network

The cluster is accessed by secure shell through a wireless local area network (WLAN). The [WL-520GU router](https://www.asus.com/Networking/WL520gU/HelpDesk_Manual/) is flashed with [Tomato firmware](http://www.wi-fiplanet.com/tutorials/article.php/3810281/How-to-Set-Tomato-Firmware-for-Wireless-Client-Modes.htm) to act as a managed switch. Another open source alternative is [DD-WRT](https://www.dd-wrt.com/wiki/index.php/Asus_TFTP_Flash).

The router can be [restored](https://chrishardie.com/2013/02/asus-router-firmware-windows-mac-linux/) if something goes wrong.

Navigating to IP address [192.168.1.1](https://192.168.1.1/basic-static.asp) with username and password "admin".

Here MAC addresses of various components can be assigned static IP addresses to make addressing easier. It can also do [name addressing with DNS](http://www.rhodesmill.org/brandon/2008/tomato-reverse-dns/).

The switch interface is then accessed by navigating to a common gateway interface (CGI) at [192.168.1.138](https://192.168.1.138/login.cgi). The default password is "password".

## Headless access

### RPi

The nodes will only be accessed as headless servers, so you may as well use the Raspbian Lite operating system. It is 1GB smaller, and doesn't come with bloatware. Download the image, and flash it to an SD card with Etcher. Drag and drop, super simple.

To enable `ssh` by default, mount the SD card (diskutil on macOS), then `touch /Volumes/boot/ssh`. Remove the disk,Connect the RPi to ethernet, insert the disk, and power on.

Whether the machine you need to configured is on the local network or Internet, use `ssh user@hostname` to log in the first time with a password. RPi come pre-configured with the user `pi` and password `raspberry`.

Find the IP address from the network admin interface, and ssh into it. If you are connecting one at a time, you can use the `raspberrypi.local` broadcast hostname. Or assign static IPs from the router.

Once you are in, run `sudo raspi-config` to enter setup mode, and do three things:

* change the password
* change the hostname
* configure wireless

For security you can disable WiFi and Bluetooth by editing `/boot/config.txt` to include `dtoverlay=pi3-disable-wifi` and `dtoverlay=pi3-disable-bt` (respectively). Save the file and `reboot`.

To enable wireless, comment out the line and `reboot`. If something is amiss, you can `systemctl enable dhcpcd.service` to start the DHCP client.

You can also overclock by editing this file (e.g. `arm_freq=800`).

If you opted for the full Raspbian experience, you can clear unnecessary packages:

```bash
git clone git://github.com/raspberrycoulis/remove-bloat.git
sudo chmod +x ./remove-bloat/remove-bloat.sh
sudo ./remove-bloat/remove-bloat.sh
```

### Digital Ocean

On Digital Ocean, log in as root using the supplied password with `ssh root@hostname`, and complete the prompts to set a new Unix password.

## Users and privilege

The first time you log in, also setup a new user account with `sudo useradd user`.

Use the `-M flag` to not add a home directory.

Then choose a strong password, and add it with `sudo passwd user`.

The user can be given `sudo` privileges with `gpasswd -a user sudo`.

## Generating keys

Next up is enabling password-less access. On your machine (the client) generate a key and copy it to the server.

```bash
ssh-keygen -t rsa
ssh-copy-id user@hostname
```

For greater security, you would normally disable root access by adding `PermitRootLogin no` to the `/etc/ssh/sshd_config` file, and restarting the service `sudo systemctl reload sshd.service`

## Package installation

The `apt-get` package manager gets registered distributions from public repositories and install them on your system.

Before installing anything new, update the registry and already installed software with,

```bash
sudo apt-get update
sudo apt-get upgrade
```

```bash
sudo apt-get install rsync build-essential manpages-dev gfortran nfs-common nfs-kernel-server vim openmpi-bin libopenmpi-dev openmpi-doc keychain nmap xinetd tftpd tftp libgdal-dev mono-runtime xsel xclip libxml2-dev libxslt-dev cython
```

## Raspberry Pi

### Network file storage

On the designated head node, you can create a root directory and make it mountable for other nodes in the cluster

```bash
mkdir /export
chown pi:pi /export/
nano /etc/fstab
```

Add `hostname:/export /export nfs defaults,rw,exec 0 0`, then edit `/etc/rc.local` to include the lines `sleep 5` and `mount -a`. Restart the service,

```bash
rpcbind start
update-rc.d rpcbind enable
```

### Remote boot image

Each device can also [boot from the same image](https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md), over the network. This means only a single image needs to be configured.

To set up a second node to boot from the main node:

```bash
echo program_usb_boot_mode=1 | tee -a /boot/config.txt
reboot
mkdir -p /nfs/client1
rsync -xa --progress --exclude /nfs / /nfs/client1
```

### Manage remotely

[Remot3.it](https://www.remot3.it/) is a way to manage a lot of remote computers through a centralized web interface.

VNC can also be used, and is installed on the RPi by default, but is targeted toward desktop users.

Each computer will need to set up following these [directions](https://remot3it.zendesk.com/hc/en-us/articles/115006015367-Installing-the-remot3-it-weavedconnectd-daemon-on-your-Raspberry-Pi).
The release version can be displayed with `cat /etc/os-release`.

If you have things setup to be command line only, and need to get to the desktop GUI to configure connections, use `startx`.

Update references with `apt-get update`.

```bash
apt-get install weavedconnectd
weavedinstaller
```

Attach to your account, and create ssh services.



Landfill,

gleaming monuments to our hyper-cool ability to

(just for fun) ruin everything,

suggested an escape beyond a map,

an endless anti-place

of absolution from sin, carbon, and the eternal waste of sun gods.

Blood, history, adventure literature told us —

This is the realm of blue collar philosophy, where character matters,

flags are matters of convenience, a landlessness of opportunity.

So we capitulated the shores for coursing brine

and an aura of ancient rime.

Thank you for ruining it too.

That was our escape - our surrender to the world.

You take the teeming shore,

but leave the waves to the dreamers.



Power of command is a theoretical measure, the ability to convert amassed subordinate potential to effective action. Much like its physical namesake, power is the product of potential difference in station and forcefulness of command. In situations of significant resistance, the balance of these factors must be managed—lest influence decay, resistance escalate, or force of command reach an extreme. Authority has a different connotation, that of applied command maintained by propriety and due respect. The two are neither mutually exclusive nor congruent. A good leader may be defined by either, but effective leadership is contingent on circumstance. There is a situation in which every leader would be impotent, and one in which even a terrible leader could thrive. The very fact that leadership has a bipolar scale signifies the complexity of command. 

Going to sea in great ships requires a measure of courage and skill, and surviving it requires luck and fortitude. The interplay of personality, station and circumstance of the principals of two tragic maritime voyages will serve as example: first mate Owen Chase and captain George Pollard Jr. of Essex, and Commander William Bligh of Bounty. Had the events surrounding the two ships not occurred, history would not mention them or their successes and failures.

Owen Chase verbally lashed the inexperienced crew of Essex at every opportunity, dispelling any illusion of romance or adventure. He showed no mercy to seasick novices and established a reputation as a ruthless taskmaster. Bloodthirsty preparations ensured that Essex was ready to kill only days out of port, although there was no quarry. He was exceptionally verbal, with an imposing physical presence. Chase had faith in himself nearing bravado, and a career dependent on outspokenness. The fervor with which he hunted the sperm whale is characteristic of a business built on the reckless pursuit of profit. At the outset, he commanded ample authority by virtue of experience and strong will. 

George Pollard instead opened the voyage with a speech asserting only that authority of the officers was ultimate. Crew described him as simple and reserved, and though he held both authority and power, he wielded them with unsure hands. In urgency the Essex sailed through a squall, severely damaging her and setting them back from the start. The dramatic shakedown cruise reflected poorly on Pollard's judgement, but did show that he could remain calm in an emergency. Pollard thought to return to Nantucket for repairs rather than press on disadvantaged, but instead set a singular precedent defining his leadership style: deference to the opinions of his officers. He was not a soft man. He exercised authority when crew complained of scant meals, and promised retribution for the malicious ignition of the Galapagos. 

Pollard showed reserve after the attack: remaining with the wreck and ensuring as many supplies were salvaged as could be carried. He stayed on Henderson as long as food was available. His determination to extract every ounce of vitality available when they had nothing stands in stark contrast to his rage that the crew would demand more when they felt slighted aboard Essex. He shows a passion for providing originally denied by the Quaker parsimony of Folger and Macy. 

After sinking Essex, Pollard would continue to give orders counter to intuition. He committed to a journey that would stretch provisions even if everything went according to plan. He maintained this plan even as whale boats made no progress toward South America, as their course brought them nearer instead to Tahiti and the Society Islands. There was no way to know the utter weakness of the starved crew when he commanded them to row out of the doldrums, but it was rash to double their rations without assurance of making progress. He lapsed into dangerous generosity as despair and hunger became more pressing. This inconsistency led to sharing supplies with other boats, who had glutted themselves while others weren’t watching. Equitable concern for the group decreased their chance of survival, and made rampant cannibalism necessary.

It is unsurprising that Pollard faltered faced with Chase’s gale of personality. Chase's impatient authority continued through much of the endeavor, seen in pressing crew regardless of despair and exhaustion, and readiness to leave Henderson before game and water were depleted. The predilection to action keeps the crew distracted and the boats seaworthy. His constancy gives difficult orders weight, and he only eases and encourages when no course is left but to hold fast to life and pray. Chase's contingent fared better after the separation of the boats, and more crew would have survived if they had parted sooner. Cohesion was important when the boats required radical repair, but the vagabond community complicated the command process. 

Chase was effective, but  far from infallible. An uncharacteristic restraint in not lancing the stunned whale doomed them to wander the Pacific. The attack could not be anticipated, but his judgement in its wake would result in many deaths. 

William Bligh was a Navy man, but the voyage of Bounty, despite the trappings of a naval mission, was largely commercial. The plan accounted for no grapeshot or sabers, and only the promise of mild excitement. Essex was destined to see more carnage than Bounty, and if successful would pay better.

Essex was sunk without warning, but they were rather more in control of their subsequent course of action than Bligh. A lack of sympathy did not lead directly to his marooning, but developed into an impassable fissure from which the chaos arose. Bligh's analogy of sailors to children indicates a crucial factor in his perceived and actual relationship with the crew. He showered them in a litany of insult, but showed a genuine concern for the success of the mission by way of the welfare of its men. 

His unconventional provisioning was intended to prevent scurvy, but created discontent. Disparity in food was not uncommon, excepting Bligh's substitutions, and the men ate as much or more than a whaler could hope for. His skylarking orders were neither appreciated for diversion nor exercise. His ban of dunking at the equatorial crossing (for safety) was loathed, though he supplied ample drink to lubricate the celebrations. He was even uncommonly reserved in ordering lashings, and was noted as having taught his men skills of the trade. 

For much of the stay in Tahiti he was necessarily liberal in the allocation of shore time. Whether the root of these allowances was pure neurotic fixation on success, or was tempered with parental concern, is immaterial. An earnest concern in shipboard affairs delivered with venom was juxtaposed with a secluded infatuation with navigational and botanical hallmarks of the voyage. He had a lingering romantic notion of the high seas, which Owen Chase would've expelled given the opportunity. The sailors aboard the Bounty were less like children than teenagers, especially in their contemptuous regard for the impositions of authority, and his privilege exacerbated this. Space was a universal inconvenience, and in all likelihood Bligh sacrificed more net comfort and privacy than the already cramped sailors. But his losses were for the advancement of an ideal and his future captaincy. Bligh had the luxury of personal project, even if he sacrificed the luxury space to the breadfruit nursery. His investment in the form of familiar personnel, financial risk, and reputation would pay dividends. In the same way that he saw every infraction as a personal offense, those under him read into his decisions self-serving exploitation.

Successful whalers reveled in the thrill of the hunt, and held stock in the financial success of the trip. This engendered healthy competition and pride in their trade. Successful naval sailors were paid so long as they managed to breath and haul line. Their preoccupations can be summarized as: get there, get back, get paid, get laid (with some ordinal variation). Harrowing experiences would be yarned, but there was no prestige or higher ideal gained by a feat like circumnavigation: career enlisted men had the rest of their lives to incrementally supplement their arsenal of tales. They did not fear Cape Horn, but Bligh's dedication and bravery in this personal goal (and an admiralty order) would have seemed misplaced and inconsequential.

Bligh was the kind of micro-manager he himself would have lamented serving under, as he resented Cook’s imposition on his duties. He was a very different man than his crew, and the mind of an 18th century English gentle-person mistook the simpler satisfactions as a waiver of self-governance. It was an unusual group only in his personal placement of associates, and typical in most regards, and simply understanding the psychology of the common sailor and their patent devotion to convention may have prevented a mutiny. He was unable to anticipate or transcend the fractionation of the crew: in this regard Bligh was not a great leader, but neither was he an exceptionally poor leader.

Bligh's thunder lacked lightning. His inaction or ignorance regarding the crew's barely covert mockery made such actions permissible, while his alteration of punitive records lessened the threat of long term consequences. The mutiny was not the product of sailors confounded by mutable social boundaries, and much less about abuse and violence. It began long before the oldest of the perpetrators or victims were born, and became more likely with every crew member born into the social stratification of the time. Tahitianization of the crew shows their tendency to rebellion against the status quo, and the the mass mutiny at Nore would soon show the ubiquity of discontent in the Royal Navy. The metastasizing discontent became disproportional to any offense committed by Bligh: he is guilty of inability to manage the escalating crisis, but not of pernicious behavior.

For general irresoluteness I would be wary of serving under Pollard. For delusional misjudgment in direction, and tendency for abuse, I would loath working under Chase. Of the three, I would prefer to serve with Bligh. He was engaged with the officers who he believed of sound mind and character, and was generally agreeable so long as he had no cause for disappointment. His eccentricity is derived from intellectual curiosity, and I believe his level of personal investment would ensure the success and safety. Furthermore he was a brilliant navigator and principled moralist. Though his character was ill appreciated by the crew for lack of mutual understanding, he had significant raw potential for leadership. The admiralty is unlikely to have sided with the mutineers in any case, but it is telling that they awarded him captaincy after clearing him of responsibility.

A chimeric blend of Chase's force, Bligh's attention to detail, and Pollard's judgement would make an ideal leader. It is unfortunate that their respective faults killed many, and we are forced to judge them through the lens of time and catastrophe.


Several reports on marine labor supply and industry in Maine came out in the recent past. 

One was written by four Scottish consultants under the guidance of Maine marine industry and research organizations. The 130-page aquaculture workforce (AWFD 2020) report solicit survey responses on the challenges business face in hiring in rural Maine, focused on qualifications and skills. 

The other is a ten-year strategic plan (MEDP 2019) for economic development for 2020-2029, which hinges on talent and innovation. Despite the grand scope, the report weighs in at 44 pages. 

Both are cautiously optimistic in the face of grim reality. If there is no systemic change, Maine is on track for a 10% reduction in labor supply by 2030, alongside a slowdown in global markets that could disrupt export industries.

The current labor force is stagnate around 700K, with a GDP of $58.5B. The report cites per worker production at $88K, compared to $120K in the US. GDP growth in Maine is 0.6% compared to 1.8% in the US. The result is that since the ’90s, pay has gone from 83% of national average to 78%, especially related to loss of manufacturing, AKA “skilled labor”.

Benchmarks for economic develop in the next decade are

- 10% increase in wages
- 10% increase in production
- 75K labor entrants

The aquaculture labor force is 600 (0.08% of the state), producing $73.4M of value (0.12%). With wages of $35.7M, the average compensation appears to be almost $60K. 

Yay! The industry is, as a whole, marginally lucrative, which in part explains why public institutions have given such vocal support for industry growth. Whether a business can bank on non-marginal investment from the state or private investors to hit these target remains to be seen, but most folks buy that it is an achievable target.

Proportional growth in aquaculture would be:

- $6K increase in wages
- $7.3M in production
- 60 new entrants

Industry interviewees suggested that labor demand would increase from 600 to 1000 (+67%) by 2030, with as many positions added to the aquaculture products supply chain. This is much greater than the state-wide target of 10%. Shellfish farmers believe that employment will not scale proportionally. 15% of businesses admitted to being revenue constrained in hiring choices, though the true number is probably higher. I say “admit”, because there are some fun secrets in the data. 

They bemoaned a lack of skilled labor, and training opportunities. Specific challenges to hiring included healthcare costs, lack of young people in rural areas, and gaps in management skills. Only 20% of the industry is employed full-time in aquaculture. A further 25% are employed part-time in other marine trades, that presumably they want to continue to do. But 50% are part-time, meaning there is actually a large available pool of trained labor. 

In fact, 42% of the industry are classified as “unskilled laborer” (by management). Less than 10% are intermediate or skilled. A full 37% of the industry is manager or director level, so manager-to-worker ratio is 2:1, where as ~10:1 is kind of “normal” in the larger world. The median income is actually around $34K, compared to the mean of almost $60K. Meaning, the owners are no longer operating, and management is capturing a significant slice of the pie. 

78% of companies are gender balanced or mostly women, but only 33% of the work force is female. So, basically, there are a minority of organizations out there that structurally exclude women. 

Companies conduct in house training for most processes, resulting in significant re-training for people rotating through multiple businesses. The report suggests that formal aquaculture-specific qualifications are the answer. They place emphasis on community college degrees, as well as opportunities to develop 2000-hour apprenticeships. The problem with apprenticeships is that for them to be approved by the state they must show career and wage progression for participants. And, there is clear lack of demand for appropriately compensated skilled labor. 

Though there are over 100 aquaculture businesses, 50% of the current work force is attributable to Cooke Aquaculture fish farms alone, and much of the anticipated growth is in shore-based salmon farms that are still in the permitting process and facing fierce community opposition.  

The points above illustrate divisions in the community. Capital vs. labor, large vs. small, engineering vs. farming, trust between companies, desire for educated individuals without a clear path for their advancement or plans to improve compensation. The industry is ripe for rationalization! 

Ultimately, growth will be driven by:

- seed supply
- availability of contract farming and processing
- community acceptance

Future development in the sector is expected to include technology adoption, expansion of RAS, investment in veterinary and husbandry research, consolidation, and supply chain adaption. Some 400-sq-foot licenses (LPAs) will become profitable, as 10% already have, and grow into full-scale farms. Provenance initiatives and systems for collaboration may also contribute to growth. 

I think, also, that growth will be limited by the bi-modal distribution of responsibility and wealth. This can be accomplished by evolving in-house functions to include skilled trades that would normally be outsourced. 

Based on experience abroad, producers tend to vertically integrate as they scale. Well-funded startups adapt this strategy from the start. As organizations grow, technical support and processes move to external or offsite division (processing, feed production). 

An interesting point made by some commenters was that IT skill were “not trainable” at small farms. Plus, over 85K locations in state do not meet ConnectME high speed standards of 25 MBPS down, 3 up, and many others have no connection. There is a need for last mile connectivity and contract IT services for farms in rural areas. Building out this infrastructure requires “consistent, predictable, and robust funding”. 

Offshore wind could add $200M to GDP, and lower the price of electricity, which is relatively expensive in Maine. This would control and reduce the cost of data and network centers, and computing-intensive engineered systems. 

With increasing connectivity and resources, new opportunities will emerge for technological innovation in small and medium-scale farms. With care, we can make sure that tools are for operators as well as owners, and that drive down the bottom line in ways that reduce toil and promote occupational sustainability for workers assuming great personal risk.



# Virtual environments

There are a lot of tools to help manage developing scientific code bases. 
Learn to use them. 
At the very least become familiar with environments, package managers, and version control.

The environment is the configuration and other software available to the program of interest. 
Packages are installed into an environment, and different environments may have different versions. 
This helps avoid issues of compatibility between different versions.

Version control and repositories help you keep a clean house. 
You can know who made changes when, and revert to old working versions if something breaks. 
Repositories keep your work backed up, and give others a place to download it from. 
For any sort of collaboration, these are mandatory. 


## Conda

Once you can login without a password from the terminal, we use `conda` and `pip` to build an environment 
that includes the necessary dependencies. The former will install the later.
Conda is an excellent tool for managing complex Python+ environments.

### x86 installation

For an Intel x86_64 architecture (e.g. MacBook, Droplet), you can download the Miniconda package with 

```bash
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh ./Miniconda3-latest-Linux-x86_64.sh
```

### ARM installation

For the Raspberry Pi 3, install ARM7 conda:

```bash
wget https://github.com/jjhelmus/berryconda/releases/download/v2.0.0/Berryconda3-2.0.0-Linux-armv7l.sh
chmod +x Berryconda3-2.0.0-Linux-armv7l.sh
sh ./Berryconda3-2.0.0-Linux-armv7l.sh
```

Logout, then back in, to ensure the binary is in your path. You create and activate a scientific computing environment,
in this case called “oceanics”, then install required python packages into the environment using conda:

```bash
conda update conda
conda create —n oceanics
conda info —-envs
source activate oceanics
```

## Pip

Pip is the short-form of Python-installs-Python. Install additional packages into the active environment with `pip install package`. 
You can also [package your own scripts](https://python-packaging.readthedocs.io/en/latest/minimal.html#) as pip compliant distributions, 
which will make it easier to port between systems or share with others. 
Pip can handle and automatically install complex dependencies. 

```bash
pip install bidict rdp pyproj pyshp pyglet geopandas mpi4py boost job_stream gdal cmake osmnx bottleneck rtree scrapy geopy shapely gridded dateutil numexpr tz bs4 html5lib openpyxl tables xldd xlwt sqlalchemy h5py lxml
```


### Distributing packages

An example directory structure is:

```
satcdf-1.0/
	satcdf/
		__init__.py
		image.py 
		processing.py
	setup.py
	test.py
```

The `satcdf-1.0` directory is the location of the version control system (VCS). 
In this case we use `git`, which will be addressed in the next section. 
The `__init__.py` file tells the interpreter that this is a module, this is the entry point. 
It contains only two lines, which import classes and functions from  `image.py` and `processing.py`,

```
from . import image
from . import processing
```

The former contains methods for loading, subsetting, and saving satellite data. 
The later contains methods for processing and analyzing the data. 
Test procedures are in, you guessed it, `test.py`. 

The `setup.py` is,

```
from setuptools import setup

setup(name='satcdf',
    version='0.1',
    description='Satellite processing library',
    url='none',
    author='oceanics.io',
    author_email='aquaculture@oceanics.io',
    license='MIT',
    packages=['satcdf'],
    zip_safe=False)
```

The package can be install from within `satcdf-1.0` with `pip install -e .`, which establishes a symbolic link, 
so changes in local files will automatically be promulgated when the scripts are run. 
Then you can just `import satcdf` in another python applications without specifying a path.

# Version Control

Paired with a repository, version control can help manage releases, track changes, and free distribute to others. 
Github is popular, but does not allow free private repositories. Github was recently purchased by Microsoft, 
if you care about that kind of thing. Bitbucket is an alternative. Both use `git` repositories. 

If are starting a new project, and have gone through the steps above to package code, 
you first need to create a blank repository on your service of preference. 
Create a new repository inside your upper level app folder with `git init`. 
Create a `.gitignore` file that contains folders and files to omit. 
Github provides [an example](https://github.com/github/gitignore/blob/master/Python.gitignore) for Python projects. 

Probably only `__pycache__/` and `*.egg-info` are relevant. Add `.DS_Store` on macOS. 
Use `git add --all` to flag all files except the ignored ones, commit it, and add a remote URL similar to above, 

```bash
git commit -a -m "New image processing repository committed."
git remote add origin https://username@bitbucket.org/aquaculture/bathysphere.git
git push -u origin master
```

If an open source project is already available on Github or Bitbucket, you need to clone or install it to you local machine. 
You can use `ssh` or `https`. The link can be found on the repository page.
 In the terminal, `cd` into a working directory and clone the project repository. 
 This clones this document and sample code into the current directory,

```bash
git clone https://bitbucket.org/aquaculture/bathysphere.git
```

Where it can be installed with `pip install -e .`. If you are making changes, you can add individual files to your local 
repository with `git add filename`, and then use `git commit -m "message"` from within the project folder. 
To change the remote path, update the origin URL, for instance:

```bash
git remote set-url origin https://username@bitbucket.org/aquaculture/bathysphere.git
```

Commit changes to a single file with message, and avoid dumping into the VI text editor, with the one-line command,

```bash
git commit -m "Updated README" README.md
git push origin master
```


Whether you use internal or external software products to manage and evolve your operations, your tool choices dictate how fast you can respond to changes in the natural, business, and regulatory environment.

This guide is intended to help you assess software used to manage data and assets on and in the water. Marine research and industry have suffered through decades of low-quality software that is hard to use, and locks people into specific ecosystems.

Moving forward, we hope marine operators will demand more from companies selling software products, and will coordinate with each other to develop and cultivate platforms for information sharing. This is an old idea, with a name, **platform collectives**.

To evaluate a product, you will need access to the full software suite, or a representative demo. You should **never make a decision without trialing it first**, as many design flaws are difficult to spot from feature lists and screenshots.

It is fine if you don’t understand some of the criteria! Just skip it, the rubric is not a quantitative tool, which is why these are true/false statements. **Existing products may only check a few boxes**.

## Getting it done

### Merits

- You can do the same things more easily that you could before
- Front-end consumes an API that can be accessed by third party services
- The product integrates with any tools you use, or plan to use
- The product or service is available offline
- You want to use the tool every day forever
- Ontology can adapt to new processes and scenarios
- Abstractions of processes and things are meaningful to operators
- Data and analytics services follow a(ny) standard

## Loving your interface

Choosing tools which do not take accessibility into mind will make it harder to hire people to use it in the future.

### Merits

- Content and visual media help understand and use the product or service
- Written materials have few errors
- Text is in a standard and consistent format
- Uses sans serif font
- The background and foreground are contrasting colors
- Red and green do not indicate important information
- Your collaborators speak the same languages as the tool
- You can imagine any employee regularly using the product
- Multi-selection inputs are easy to understand
- Form inputs are validated before submitted
- Labels and placeholders provide useful guidance
- Text forms allow auto-complete and fuzzy matching

## Keeping your secrets safe

Software can be your competitive advantage, but it can expose to risk and invalidate practices you used previously to maintain trade secrets.

### Merits

- SSL is kept up to date
- Data are encrypted at rest and in transit
- Registration requires a strong password
- Passwords are only ever accessed by the account holder
- Provides two-factor authentication
- Your rights to your data are clear
- The developer will not own copies of your data without consent
- Provider makes it easy to ex-filtrate your data or stop service at any time for any reason
- There are no back doors for the developers

## Getting help

Read the manual. Can you do self-service troubleshooting based on the technical and support documentation alone? Maybe there is a user community that you can go to for extra help. The truth is though, no everyone is outgoing or has the time to track down info. In those cases, you'll want to be able to reach support. If support is not in your language or time zone, or if you are a small fry that doesn't warrant the trouble, you'll be on your own.

### Merits

- Support is available in your time zone
- A team supports the product
- Support is provided by the creator of the product
- Support cost is transparent
- The company uses formal code review and quality control
- Source code is available
- Documentation includes a diagram or explanation of how database models are related
- There is self-service documentation, external to the application

## Respecting your rights

Read the license, or the End User License Agreement. It's probably a standard thing. For a mobile application that costs a few dollars that is OK. For bespoke software costing hundreds or thousands of dollars per month, that is going to help run your operation and keep you people safe, it is entirely appropriate to negotiate the terms.

### Merits

- You own the data you put into the system
- You can make changes to the code
- You can provide secure access to data for third parties
- Terms are governed by accessible courts
- The reverse engineering clause will not prevent you from using or making tools you need
- Non-compete clauses have geographic and time boundaries

## Working in the field

Maps and charts are an important part of life on the water. Tools with good geospatial data visualization features will make you life easier. Most mapping software is intended for 2D land applications.

### Merits

- The style of the map is adapted to a marine setting
- Satellite and aerial imagery is off by default
- You can find locations with and without coordinates or names
- The projection of the web map is not distorted at your latitude
- There is no additional cost for connectivity
- Runs on any mobile device in the field




## Objectives

The following introduces some fundamentals of neural networks, stuff that has been around forever and has nothing to do with TensorFlow. 

This is also not specifically about “deep learning” style inference. That is the world of regularized multi-layer perceptrons called convolutional neural networks, with each neuron connecting to all of those in the following layer.  Useful for image classification, but suffer from over-fitting. 

Rather, the desired functions are:


1. Correct user input and documents so that names and terminology are consistent, and provide learning loop to reinforce positive behaviors 
2. Infer whether objects in an abstract graph are related, and if so build relations between them and propagate changes up to aggregates
3. Make recommendations on data sources for specific uses, and identify opportunities to improve synthetic data sets
4. Predict time evolution of systems, and fill time and space gaps in sparse data
5. Parse natural language into structured queries

Most machine learning activity in marine science that I know of focuses on prediction (#4), or some type of image classification, which is not a focus here. Relationship inference and recommendations (#2 and #3) are birds of a feather, and are not uncommon applications. Natural language stuff (#5) is of course very popular. User feedback learning (#1) is extremely powerful, if you give it enough control over the presentation and operation of the application.


## Data structure

A neural network is made up of layers of neurons and synapses. People use “layers” to mean both neurons and synapses layers, which is wicked confusing. Neuron layers are floating point or integer vectors, and the synapses are 2D arrays with indices that map neighboring neurons to each other. Let’s call them neuronal `layers` and synaptic `matrices`.

Layers are instantiated with a default value of zero. The data structure used must support basic array arithmetic, as well as common functions like `max_index()`, `distance()`, and `normalize()`. Matrices must also support matrix arithmetic, instantiation from tensor operation ($M=AB^T$), and the ability to slice and insert vectors. 

The high-level interface is the network structure itself, which handles training neurons, and ushers new inputs through the system. Networks infer relationships between vectors, and may use a vector-pair abstraction in `encode` and `recall` transfer functions. Assignment and equivalence operators are required for this. Networks are defined by the size of input and output patterns, which are arguments to `encode`/`recall`, and have properties of learning rate, signal decay, and error tolerance. 

The required methods for a basic implementation are:

1. `train()` — run vector-pairs until tolerance is reached
2. `cycle()` — pass vector-pairs and invoke `encode()`
3. `test()` — testing is good
4. `run()` — run the model
5. `dump()`/`load()` — cache the synapse weights
6. `encode()` — store a vector pair in the network
7. `recall()` — complete the vector-pair input


## Learning

### Activation
Neuron internal mechanisms include an adder function which combines the weighted strength of the signals, and threshold function that then computes the activation state. Neurons emit a signal related to the activation state, which may connect to any number of other neurons. 

Activation functions are nonlinear. Using a linear function in multi-layer (deep) networks would have no advantage over a single-layer one. The most common are basic step functions and sigmoid squashing functions:


> $x=\sum_{i=0}^{n} {A_i w_i}$
> $f(x)={{1}\over{1+e^{-x}}}$
> $f(x)=tanh(x)$

### Weights
Synapses store information as weights, trained by presenting patterns in sequence. The learning law updates the weights based on the learning rate, $\alpha$. Some simple forms include Hebbian correlation learning (Hebb 1949), with the weight updated by connected activation values, or their sigmoid transform:


> $\Delta w_{ij} = \alpha a_i a_j$
> $\Delta w_{ij} = -w_{ij} + f(a_i)f(a_j)$


The learning rate is “cooled” when it is initialized as a high value, and decreases to zero over successive cycles. This is not used often in practice.

### Supervision

When there is no feedback regarding the quality of the prediction, the process is considered unsupervised, and therefore self-organizing. Modes of unsupervised learning include additive matrix learning (e.g. bidirectional associative memory) and learning vector quantizer (counter-propagation)

Supervised learning is more common. Weights are adjusted based on the error between truth and predicted values follow an error correction law such as $\Delta w_{ij} = \alpha a_i [c_j-b_j]$, where $c$ is the desired activation. 

Reinforcement learning is a form of supervision. Each output neuron gets an error value$\Delta w_{ij} = \alpha(\nu - \Theta_j)e_{ij}$. Here $\nu$ means total error and $\Theta$ means threshold. The error is disbursed back to neurons, which have eligibility function, $e_{ij}={{d ln g_i} \over {dw_{ij}}}$, where $g$ means probability of correctness. This adjustment is based on activation state, and does not depend on changes during previous step.

Additional hidden layers makes patterns of higher dimensions separable in parameter space. Training is more complicated, and involves back-propagation of error through hidden layers, a is the weighted sum of errors in the output layers times the activation of the hidden neuron.




## Back-propagation

Back-propagation (Rumehart and Hinton 1986) is the predominant method of supervised learning. It requires a lot of training data, and domain knowledge of the ideal number of layers and terminal neurons. 

The size of hidden layers should be between the number of input  and output layers. 
The model tends to group features from the domain, such as pixels forming line features in optical character recognition. Inputs must be normalized and clamped to (0, 1), and the weights can be adjust either incrementally or once per epoch. 

### Encoding

Weights of synapses and neuron thresholds are randomly initialized in (-1, 1). Encoding vector-pairs is accomplished in two stages. The forward pass computes the hidden layer activation, $h=f(iW_1)$, and then the output activation, $o=f(hW_2)$. $W$ means synaptic matrix. The backward pass computes the output error $d=o(1-o)(o-t)$ from a known target $t$. This is used to update $W_2$: 


> $W_2 = W_2 + \Delta W_2$
> $\Delta W_{2,t} = \alpha h d + \Theta \Delta W_{2,t-1}$

Then the hidden layer error, $e=h(1-h)W_2d$, updates $W_1$:


> $W_1 = W_1 + W_{1,t}$
> $\Delta W_{1,t} = \alpha ie + \Theta \Delta W_{1,t-1}$

Cycle until $d$ is within tolerance.

### Recall
The network recalls by computing hidden ($h$) and output ($o$) activation as before. Thresholds may be used to bias activation,

> $h=F(iW1 + thresh1)$
> $o=F(W2h + thresh2)$
> $thresh1 = thresh1 + \alpha d$



## Counter-propagation

Counter-propagation networks (CPN) train faster than back-prop, and are better generalizers of new data which may lie outside distributions previous encountered (Hecht-Nielson 1987). The method combines a linear vector quantizer (LVQ, a form of self-organizing map) and a minimal outstar encoder.  These can function as lookup tables. These are generally unsupervised, but may be amended with feedback.

There are three layers, and the hidden layer usually only has a single neuron activate at time (some cases there may be more). This is chosen by similarity to encountered patterns, and is a categorization process. There must be a neuron in the hidden layer for each output neuron (category), which is referred to as a capacity problem.

### Encoding
The initial weights are random. Inputs are chosen from the sum of squared differences. Weights of connecting synapses are updated from the difference between input and current state. The outgoing synapses are then updated from the single activated neuron.

### Recall
New queries activate a single hidden neuron, which produces an output pattern indicating the category.


## Bidirectional associative memory systems

The capacity problem of CPN is solved with bidirectional associative memory systems (BAMS). The original formulation (Kosko 1988) was improved upon by Patrick Simpson (1989), by adding new matrices of synaptic weights when the current ones saturate. 

This model is based on two-layer feedback nets used for memory addressing, and has useful properties of being stable (training will always converge) and having instant recall. The problem is that the memory usage increases O(n2) with pattern length.

BAMS are useful when mapping between features:

- optical character recognition: pixel pattern —> character
- voice: signal —> word
- spell check: words —> phonemestring
- radar: signal —> object


### Encoding
The matrix is the sum of correlation matrices of pattern pairs,


> $M= \sum_{i=1}^{m} A_i^T B_i$

New pattern pairs are added to the current encoding by computing incremental additions. Pairs can be removed by performing the inverse operation. The matrix is tested to see that each pair can be recalled. If not, the last entry is removed and stored in a new matrix.

### Recall

When one part of a pair is presented, the network will recall the complement, or a close match.
Patterns are passed back and forth through the matrix and transpose until it converges on a solution, which is guaranteed to exist according to Lyapunov energy function,


> $E(A,B)=-AQB^T$

Don’t know the location of the information, since it is spread around, so the program queries each matrix for pairs with the same energy as desired. This is more computationally expensive than checking a single source. 

By modifying the M encoding methods, can favor convergence or accuracy. Options include the Hebb’s signal function from before, competitive encoding, or differential Hebb’s. 


## Hopfield Nets

Single layer auto associative pattern storage (Hopfield 1985) useful for applications like the traveling salesman problem. 

Encoding 
For discrete cases, add product of activation to weight, 


> $W=\sum A_k^T A_k$

Continuous nets are more flexible, act like circuit, with neuron conductance,


>  $w_{ij} = f(a_j)/R_{ij}$

The function is arbitrary.


Research organizations have observing platforms to deploy to aquaculture growing areas. The annual amortized cost can be $100,000. To ensure that resource allocation to the private sector is rational, we automate co-location agreements through a weighted lottery. We call this a `Campaign`. 

One campaign could measure ten observed properties important to siting and operating at hourly intervals: temperature, salinity, wind speed and direction, air pressure, precipitation, sunlight, water speed and direction, pH, chlorophyll, and turbidity.

Public notice goes out to businesses and organizations, and an online application is available for at least 2 weeks. This consists of a web form soliciting proposals to site one or more assets.

<a href="assets/observer-lottery.pdf">You can see a visual representation here</a>.

You represent an aquaculture production business in good standing with the state of Maine, and have a lease, license, or pending application. You submit decimal degree coordinates in waters less than 100’ at mean high tide within the region defined by coast, Portland Head Light, Potts Point, and Half-Way Rock. 

The data we collect are: 
- `e-mail`, of course
- `company`, for determining eligibility
- `contact-consent`, make sure we can follow-up
- `latitude`, latitude
- `longitude`, longitude
- `observed-properties`, # of properties that the agent observes
- `locations`, # of locations where the agent observes
- `data-stream-period`, rate of observations

Groups of applicants can submit duplicate locations, and these will be ranked higher. Our accounting system is based on observation utility increasing when:

- Dense but not overlapping
- Filling gaps in coverage or frequency
- Usable by many operations
- Fully utilized

We use three functions: `verify(agent)`, `cluster(agents, n)`, and `rank(cluster)`. 

To avoid skewing selection, we don't release parameters until after a final selection, and when any agent is found ineligible we restart. Selection continues until there are there are no more agents, or an awardee agrees to inspect the buoy during normal operations and report damage or loss in a timely manner

We form negotiating teams using fuzzy C means (e.g. Dunn 1973) on location and aggregate agent-reported effort for the group. Agents with no current efforts are not able to apply.

The team is ranked using a logistic function of information gain provided by co-locating infrastructure. To avoid gaming the system, we don't release gain parameters until after a final selection. Your effort is `1/period * observedProperties * locations` and the value of the service is 64,800 observations. 

One team is picked randomly with the probability proportional to rank. The member with the greatest reported effort is the designated representative. Agents are notified when selected, withdrawn, or the process ends. 

Final location is at the discretion of the owner and deploying vessel. The owner maintains infrastructure and makes sensor observations public on the web within one hour without guarantee or warranty for suitability for any purpose. The owners and Oceanicsdotio LLC use anonymous data for market analysis and research reporting.

The journey of a molecule related by Aldo Leopold 
is an exercise in poignant prose; 
but here, if I may be so bold, 
I make an effort to transpose 
his Odyssey, to the lexicon of biogeochemistry. 

And since X is no fitting name for this history, 
we will equally consider she 
as carbon and as nitrogen, 
and trace the path of N and C 
from limestone cliff through cyclic sea. 

As NaNO3, nitrogen has well-known solubility. 
and natural fertilizing ability; 
but for those who prefer C 
the mineral source can be 
easily, ubiquitous deposits of CaCO3. 

First a plant took root up there, 
and split the stone and let in air 
bearing H2O and CO2, 
from which carbonic acid grew 
(with the carboxylic additions of oak root too), 

which weathered the rock, from whence the bur-oak drew 
a solution of nitrate 
some calcium bicarbonate, 
which when assimilated may 
eventually have constructed a strand of DNA. 
 

So were exhumed our fated friends, but to what end: 
victim to assimilatory metabolism 
and kept interred in organisms. 
Turn by turn they onward flowed 
until such time as their owner decomposed. 
 

Through each phase of this biological pump, 
from plant to beast to human race, 
both N and C keep different pace: 
depending on their placement 
as reactant or cellular cement. 
 

When an organism in which they'd happen to reside 
would finally succumb to time's tide, 
came then the turn of fungi 
and hungry bacteria to ammonify, 
and percolate the soil with carbon dioxide. 
 

Then a root would re-uptake, to assimilate in a leaf, 
before a mouse physically bore them beneath 
the surface realm of wind and rain 
to there delay another refrain. 
And while N and C wait to again respire 
 

biological cycling is halted by wildfire: 
the combustion of biomass 
oxidizing N and C en masse, 
and losing N2 and CO2 to air 
but storing charred organic matter beneath the surface soot. 
 

With the return of legumes and their bacteria-laden roots 
nitrogen fixation from atmospheric 
to NH4+ (always anaerobic), 
which in turn was nitrified, to NO3- 
which fueled respiration, and C again was free. 
 

So began a race, hydraulically transported by slope 
with impetus again 
derived from rain, 
delayed by cyclic residence 
in all manner of heterotroph. 
 

The seaward rush of N and C progressed until they found 
themselves united in urea or aminos; 
after sulphate reduction in anoxic silt 
and dissimilatory metabolism, they plunge into 
(presumably) the Gulf of Mexico. 
 

This entirely new system was not for Aldo to explore. 
To cycles in the ocean, more pressing than on land, 
he gave the brand 
of prison, with which I disagree: 
for no system is a prison in terms of biogeochemistry. 
 

While passing through a gateway estuary 
the nutrient spiral roulette 
could spit into the Atlantic 
either inorganics, 
or as here presumed: waste organic matter. 
 

DOC is dominant, one fifth of all the latter. 
N is surface bio-limiting, 
while C is bio-intermediate, 
both can be dissolved or particulate: 
a functional difference in size, meaning that one sinks 
 

while the distribution of DOM is circulation driven. 
When in the euphotic zone 
most forms of OM are, prone 
to surface photochemical reduction, 
and NH4+ driven regenerative production. 
 

Though rates vary by geography, N remineralization 
occurs, first by ammonium-nitrate oxidation, 
followed by denitrification 
(to solute N2 and H3PO4), 
then by metal ions, and sulphate reduction (of which I'll mention more). 
 

The generated biomass of net primary production, 
feeds a chain of organisms 
with less efficient metabolism, 
this bio-pump trophically concentrates 
once-dissolved OM into heavy fecal particulates. 
 

But this and other sinking POM will not go unused: 
at aphotic depths it is reduced 
to CO2 and nutrients 
by heterotrophs reliant 
on settling, and the vital circulation which upwells once more. 
 

Then the NO3- and CO2 are used by coccolithophores, 
to fuel new production of their plates 
from Ca2+ and bicarbonate, 
and when the algae die, the CaCO3 
quickly settles to benthos and becomes sedimentary. 
 

When I consider that only a quarter of OM 
(regardless the sort) 
makes it to the ocean floor 
by settling or transport; 
and 90% is re-mineralized for a few reactions more, 
 

I can say with confidence, that the chance is rather small, 
that even through many iterations 
they'll see the abyssal plain at all. 
So, though C and N may languish 
in sediment and solution and fish 
 

it's never so long, speaking geologically, 
despite the volume of the sea, 
before carbon and nitrogen 
are returned to air and land 
through crustal subduction and steady ebullition.

I made some predictions about the continued expansion of aquaculture in Maine. 
Now that we have 2 more years of data and are 40% through a **5-year forecast**, 
let's look back at the early concept: *for long-term predictions, 
you need to model human behavior, not just the environment.*

If you are unfamiliar with the area, the coastal town jurisdictions are shown in red below.

![Maine coast](maine-coast-bw.png)

Maine farmers cultured \$6.7M in mussels, oysters, and clams in 2015, and $8M in 2016 (Maine Department of Marine Resources): e
xceeding federal growth benchmarks [1]. Leasing is **decentralized and reactive**, with exclusion criteria including water uses, 
public assets, and lease density. Larger operations improve efficiency, and attract processing facilities—but there is 
concern about the size of firms [2]. Residents closest to farms view them as locally undesirable [3], and decentralized 
expansion — including limited-purpose licenses (table below) — is unlikely to balance production with social acceptance.

| | Standard lease | Limited-purpose license |
|----|----|----|
| Hearing | Yes | No |
| Area | 0–100 acres | 400 sq. ft. |
| Term | 10 years | 1 year| 
| Density | None | 4 per 1000 ft. radius |

Sustainable growth instead requires evaluating how **scale** matches **community objectives**. 
Targeted **non-marginal investment**, coordinated through voluntary bay management [2], can increase income and equity.

In **single-bay management** (SBM), industry cooperatives unify siting and culture practice. 
Objectives are to lower expenses, and fix or increase price. New sites have a mean frequency ($ λ $, y-1), 
such that new arrivals follow a Poisson counting process. Damariscotta River had 91 LPAs as of 2017. Issuance is 191 y-1, 
accelerating by 30 y-2 (DMR). Spacing restrictions give a footprint around 0.024 km2. 
Given a window of size $ A_0 $, and a set of $ N $ existing leases, no viable space will remain 
in jurisdiction area $ A $ after $ t=A(41.66A_0-N)(\lambda A_0)^{-1} $. 

A less ideal packing efficiency that accounts for operators buffering themselves is 0.093 km2, which significantly reduces these estimates. 

Growth may vary across tens of meters, so siting is fine-scale (figure below). New locations are sampled from a kernel density estimator, 
trained on existing locations, depth, and satellite-derived oyster suitability. With projected growth, there are >200 sites in the region 
by year 3 (table below). Mean $ t $ is 15 years, without considering competing uses or closures. Less than 22% allocation, 
and some jurisdictions are already full. The model makes assumptions about space and market conflicts, given the area has a unique aquaculture history.

| Year |$ N $|$ \lambda $|$ t $|$ min(t) $|
|---|---|---|---|---|
| 0 | 91 | 36.4 | 15.2 | 0.001 |
| 1 | 127 | 37.0 | 14.9 | <0 |
| 2 | 164 | 37.6 | 14.5 |<0 |
| 3 | 202 | 38.2 | 14.1 | <0 | 


One alternative is **site allocation planning** (SAP), which delineates growing zones [5] by environment and conflicting uses 
(e.g. Maryland Natural Resource Code 4-11A-05). Production can be optimized within a given footprint. 
Area incentives like credits or subsidies are structured at or above the municipal level—changing $ λ $ within a 
jurisdiction—while shore facilities create point amenities. **Multi-stakeholder boards** (MSB) seek granular 
siting input, and increased municipal involvement [2]. This aids information gathering, and increases legitimacy [6].

<img 
 src={"https://oceanicsdotio.nyc3.digitaloceanspaces.com/bathysphere/images/damariscotta.tiff"} 
 alt={"Map of oyster suitability index and limited-purpose licenses in the Damariscotta River estuary, Maine. High value areas are saturated, and continue to infill in predictions."} 
 class={"wp-image-218"}
/>
<figcaption>Damariscotta River showing oyster suitability index (OSI). The field is a quadratic reconstruction of raster data on a triangular mesh (R2=0.72). Red is existing limited-purpose licenses (N=91); blue predicted locations (N=100) based on SBM assumptions. 
 
 Data: ME OIT, UMaine, Oceanicsdotio
</figcaption>

Potential drawbacks include exclusion, and prolonged process. Neighborhood attitudes as expressed at hearings [3] can 
be used to create probabilities of opposition. This approaches zero when a community has few farms, complaints, or low 
willingness-to-pay. While towns and regional co-management groups **shall not** exclude aquaculture, 
they can engage in planning exercises to determine how siting can be structured in a desirable way through manipulation of
 the amenity field ($m-2y-1).
 
This is just one of many methods we use to develop strategic plans for water quality management.

1. NOAA Marine Aquaculture Strategic Plan FY 2016-2020. 
2. Governor’s Task Force on the Planning and Development of Marine Aquaculture (2004).
3. Evans et al. (2017). Ag. &amp; Resource Econ. Rev.
4. Snyder et al. (2017). Fron. in Mar. Sci.
5. Sanchez-Jerez et al. (2016). Aqua. Env. Inter.
6. Cash et al. (2003). SSRN E. Journal.


## Right-triangulated irregular networks

The numerical simulations we use are executed on triangular meshes or multidimensional arrays (aka "raster" or "texture" data). For optimizing visualization and on-the-fly calculations in the browser we instead use specialized meshes like the right-triangulated irregular network (RTIN).

This is a hierarchal data structure for representing a regular rectilinear grid as a triangulation. For the purposes of visualization, the height values at the grid points are assumed to be exactly correct.

This is a form of multi-resolution surface rendering which forms right isosceles triangles from a subset of the points. Multiple partitioning schemes within the representation allow for changing the resultion dynamically.

The algorithm to decompose a square into triangles is:

1. first divide along NW-SE
2. form partitions by splitting triangles larger than the minimum size
3. split T from right angle to midpoint of hypoenuse
4. if edge point causes neighbor (R) to become a quad, propagate
5. if equal size stop, else if R larger continue to propagate

This is also called a `4*8^2` Laves net. Laves nets are tessellation methods where every subdivision has a similar shape. The numbers are the maximum splits that occur along each side of the reference shape. Squares are `4^4`, 30-60-90 triangles are `4.6.12` and equilateral triangles are `6^3`.

Squares and equilateral triangles cannot form a continuous non-uniform partition, because any split with recursively divide all cells. The `4.8^2` will change at most 2 of each size triangle, while `4.6.12` change 12 or fewer of each equal and larger size.

In practice, this is implemented as a binary tree, with triangles as leaves. The root node is the square. Each half of a split polygon is labelled `left`/`right` according to the side of splitting ray that it is on.

Splits are from the hypotenuse to the `right` vertex. From a parent ordered counter clockwise with the right-angled vertex labeleld v_3, the `left` partition is (v_3, v_1, m), and `right` is (v_2, v_3, m).


We use a combination of databases and custom software to do geospatial analysis. This document provides some examples of data processing and ingestion to do point-based queries for data.

If you want, you can get the large (>1GB) NetCDF file here:
https://oceanicsdotio.nyc3.digitaloceanspaces.com/LC8011030JulyAvLGN00_OSI.nc

This file contains oyster suitability data aggregated over one month. It was synthesized by Jordan Snyder at the University of Maine.

We'll want a basic table of point observations:
```sql 
CREATE TABLE landsat_points(
 longitude DOUBLE PRECISION NOT NULL,
 latitude DOUBLE PRECISION NOT NULL,
 oyster_suitability_index DOUBLE PRECISION,
 geo GEOGRAPHY
);
```

For simplicity, you can unpack NetCDF into CSV and use bulk imports. This is pretty easy in JavaScript and Python, but non trivial in, like, Fortran. You can end up with big files doing this. Split the file, and wrap the ingestion process as a bash function:

```bash
echo Splitting file...
split -l 2000000 xyz.csv xyz-;
parts=$(ls xyz-*);
echo Uploading ${parts}...
count=0;
for part in ${parts}; do
 if [[ ${count}==0 ]]; then
 yourBashFunction ${part} HEADER;
 else
 yourBashFunction ${part};
 fi
 count=$((count + 1));
 rm ${part};
done
```

Your ingestion function probably looks something like:

```bash
PGPASSWORD=${PG_PASSWORD} psql \
 --echo-queries \
 --username=${PG_USER} \
 --host=${PG_HOST} \
 --port=${PG_PORT} \
 --dbname=${PG_DATABASE} \
 --command="\copy landsat_points (longitude, latitude, oyster_suitability_index) FROM '$1' WITH DELIMITER ',' CSV $2"
```

Then you have to create `postgis` geometry from the lat and long, and build a spatial index on this:
```sql
UPDATE landsat_points
SET geo=ST_SetSRID(ST_MakePoint(longitude, latitude), 4326)
WHERE geo IS NULL;
CREATE INDEX landsat_points_geo_idx ON landsat_points USING GIST(geo);
```

If you are pulling from an ESRI shapefile, learn `shp2pgsql`. This can even be used for a remote source. 

Once the points are ingested you do a bounding box query :
```sql
SELECT * FROM landsat_points AS points
WHERE st_within(geo::geometry, 'SRID=4326;POLYGON((-69.6 43.8, -69.5 43.8, -69.5 44.1, -69.6 44.1, -69.6 43.8))'::geometry);
```

Or, average nearby observations:
```sql
SELECT AVG(osi), COUNT(osi) FROM (
 SELECT osi FROM (
 SELECT oyster_suitability_index as osi, geo
 FROM landsat_points
 ORDER BY geo <-> 'POINT(-69.89196944 43.77643055)'
 LIMIT 24
 ) AS knn
 WHERE st_distance(geo, 'POINT(-69.89196944 43.77643055)') < 60
) AS points;
```

Or, get nearby observations:
```sql
SELECT osi, geo, st_distance(geo, 'POINT(-69.89196944 43.77643055)') as dxy FROM (
 SELECT oyster_suitability_index as osi, geo
 FROM landsat_points
 ORDER BY geo <-> 'POINT(-69.89196944 43.77643055)'
 LIMIT 24
) AS knn
WHERE st_distance(geo, 'POINT(-69.89196944 43.77643055)') < 20
```

With millions of points you can still resolve queries in under 100ms on a modest cloud Postgres instance!

I want to provide a high level view of some statistical methods and data structures we use in our work.
These are meant to be living notes, and may contain stubs or incomplete information. There are plenty of exhaustive resources, but I recommend Matteo Matteucci's
<a href={"https://home.deib.polimi.it/matteucc/Clustering/tutorial_html/index.html"}>work</a>.

K-means (or medians) is an *exclusive* model that associates observations with clusters.

The number of clusters cannot be determined analytically. Instead $k$ is chosen based on domain knowledge and refined iteratively according to the Schwarz Criterion. The model is sensitive to initial positions, so the choice matters, and Monte Carlo replication should be used, otherwise it tends to find local maxima. The steps are:

* distribute clusters uniformly in multi-variate space
* assign observations to their nearest cluster
* recalculate cluster centroid from children
* repeat until clusters are stationary

This process minimizes the squared error function:

$ J = \sum_{j=1}^{k} \sum_{i=1}^{n} || x_i ^ j - c^j ||^2 $

Hierarchical models view clustering as a binary tree of most-similar observations. This can be built top-down (divide) or bottom-up (agglomerate).

1. compute (half) distance matrix
2. create pair cluster from indexed observations
3. repeat until there is a single top-level cluster

The final product is a b-tree implemented as a hash map. The distance function is a domain specific comparison that logically connects clusters by their most similar (single), least similar (complete), or mean (mean).
Some additional hierarchical resources with more detail:
[1](https://www.displayr.com/what-is-hierarchical-clustering/)
[2](http://www.cs.princeton.edu/courses/archive/spring15/cos233/clustering2.pdf)
[3](https://www.wave-access.com/public_en/blog/2015/november/17/memory-reduction-for-average-link-hierarchical-clustering-algorithm-with-a-large-amount-of-input-data.aspx)

Fuzzy clustering is an overlapping model, in which observations have a probability of belonging to every cluster. A common approach is fuzzy c-means (FCM), which extends k-means models, and uses the centroid method.

Model-based methods are probabilistic. Clusters are represented by mixing a collection of Gaussian or Poisson components.

These are some of the canonical references in the field:

1. MacQueen, J (1967). Some methods for classification and analysis of multivariate observations. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability (pp. 281–297). Berkeley: University of California Press.
2. Johnson, Stephen, C (1967). Hierarchical Clustering Schemes. Psychometrika, 32(3).
3. Day, WHE, & Edelsbrunner, H (1985). Investigation of proportional link linkage clustering methods. Journal of Classification, 2, 239–254.

## Concept 
In a perfect world, I would know the tides before going out on the water. But I forget all the time. 

I've worked with teams that operated entirely around the tides, and others that operated regardless of tides. In the Northeast the tides are a constant feature, while in Louisiana they barely exist.

The historical data are useful for providing context to events. Tide gauges are also more generally water level gauges, which mean that they include storm surge and wind-influenced levels.

Real-time data is an aid to navigation.

Future conditions can help plan operations and assess risk. Let's no get into sea level rise though.

## Source
US water levels are available from [NOAA Tides & Currents](https://tidesandcurrents.noaa.gov/stations.html?type=Water+Levels#Maine). Your tax dollars at work. 

There are 6 stations in Maine. These generally also have metocean data associated with them, like wind, air pressure, and water temperature. 

Lucky for us the Tides & Currents API is [documented](https://api.tidesandcurrents.noaa.gov/api/prod/)! Unfortunately, the responses are not really self-describing. "What do the letters in the data columns represent?" asks their own [help](https://api.tidesandcurrents.noaa.gov/api/prod/responseHelp.html) page. Great. If you feel defensive about your own API design, you might want to reconsider your choices. 

Doesn't seem like you can request with a geographic range.

Looking at the network requests trigger by a map load, you can see that all the stations are retrieved from a metadata API (`/mdapi/prod`), and then each is populated with individual requests to the core API (`/api/prod`). FYI, it's SOAP/XML underneath a JSON friendly wrapper. 

The [docs](https://api.tidesandcurrents.noaa.gov/mdapi/prod/) for the metadata API indicate that yes indeed you can search by radius! 

But, only when requesting a specific station. In other words, you can return `nearby` stations as part of the request for a named station. Useful for comparisons and topological queries.

So, we absolutely must get all stations, and then filter them in the client. Fine NOAA. They "expand" linked data by default, but let's not do that. This initial request only takes 55 ms, and can definitely be cached. But it also moves 600KB of data just to get references to 11 links that I need to follow. 

The docs describe all of the query parameters for fetching the actual data. 

There are many time interval options. Some time based query parameter is required. Use `date=latest` for a single observation, or `range=24` for past day. 

You also have to explicit declare a `product` value or array, `datum`, `time_zone`, `units`. Let's use Mean Lower Low Water (`mllw`) cause it's the default. We'll go with daylight savings local time, and metric.

They want you to provide information about the accessing `application`. This is good form, gotta justify providing public data with tax money.

The final thing is:
```
https://api.tidesandcurrents.noaa.gov/api/prod/datagetter?date=latest&station=8454000&product=water_level&datum=mllw&units=metric&time_zone=lst_ldt&application=oceanics.io&format=json
```

## Using with Mapbox and React

Our stations query (as a React hook) is something like:

```TypeScript
const EXTENT = [-71.190, 40.975, -63.598, 46.525];
const metadata_url = "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.json?type=waterlevels";
const data_url = "https://api.tidesandcurrents.noaa.gov/api/prod/datagetter";

const cropToExtent = (extent) => ({lat, lng}) =>
 lng >= extent[0] && lng <= extent[2] && lat >= extent[1] && lat <= extent[3];

useEffect(() => {
 map.addImage(id, ANIMATED_ICON, { pixelRatio: 4 });

 (async () => {
 const {stations} = await fetch(metadata_url).then(r => r.json());
 const queue: Promise<object>[] = stations
 .filter(cropToExtent(EXTENT))
 .map(({id}) =>
 fetch(
 `${data_url}?date=latest&station=${id}&product=water_level&datum=mllw&units=metric&time_zone=lst_ldt&application=oceanics.io&format=json`
 ).then(r => r.json())
 );
 map.addLayer({
 id,
 type: "symbol",
 source: parseFeatureData({
 features: await Promise.all(queue), 
 standard: "noaa"
 }),
 layout: {"icon-image": "tidal-stations"}
 }); 
 })();
}, []);
```

We're waiting for the Mapbox data structure to load, as well as some HTML canvas based icons to create a custom symbol layer on the map.

Station metadata is retrieved then filtered down to only the stations within a bounding box. This could be any spatial query!

The selected stations are queried and then these are parsed into a GeoJSON FeatureCollection embedded in a GeoJSON layer Mapbox source object. The parsing function `parseFeatureData` is omitted here. The `ANIMATED_ICON` is a canvas routine to draw a changing level indicator. 

The problem is that you only get one datum per request, like Mean Lowest Low Water (`mllw`), but you need two to be able to render a progress-bar style of gauge. 

So another request the `datums` product for every location before you can paint the map. Blech!

## Trie

Tries are graphs of symbols. Words are accumulations of symbols along a path into the graph.

You `insert` words into the network, and mark the nodes that terminate words. All words sharing a prefix traverse the same path.

You can then `search` the trie for words, e.g. similar to a key word. This can be used to [correct spelling](http://norvig.com/spell-correct.html) and auto-complete inputs.

That requires a similarity function (Levenshtein distance). You traverse the graph and iteratively build a N by M table, “answer” is the lower right value. This process is O(N\*M).

Tries can waste memory or storage. [The lower limit is 4 bytes per edge](http://www.wutka.com/dawg.html), while remaining searchable.

For a vocabulary of ~100K words, a naive JSON output of the structure is 8MB.

We use tries to try to auto-correct fields in forms and API requests, provide helpful suggestions when there are bad inputs, and to clean and restore strings that have invalid characters.

### Variants

- When words share a prefix, you can extend the table instead of recomputing, which happens automatically given alphabetical inputs.

- Directed acyclic word graphs (DAWG): more efficient due to similar words with different prefix converging. These are also known as [minimal acyclic finite state automaton (MA-FSA)](https://www.aclweb.org/anthology/J00-1002.pdf). Insert alphabetically, and deduplicate after each branch. Duplicates are XNOR ends and point to the same nodes.

## Distance function

A direct comparison of words calculates the number of inserts, deletes, and replaces that need to occur for the sequences to be equal.

A minimal `JavaScript` version looks like this:

```JavaScript
/**
 * Calculate similarity of two patterns, usually words
 * for the purpose of auto-correct or spell checking
 */
const LevenshteinDistance = (a, b) => {
 let row = [...Array(a.length + 1).keys()];
 
 for (let ii = 1; ii < b.length + 1; ii++) {
 const previous = row;
 row = [previous[0] + 1];

 for (let jj = 1; jj < a.length + 1; jj++) {
 row.push(Math.min(
 row[jj - 1] + 1, // insert, 
 previous[jj] + 1, // delete, 
 previous[jj - 1] + (a[jj - 1] !== b[ii - 1] | 0) // replace
 ));
 }
 }
 return row.pop();
};
```

The observant reader will notice it has been a fucking minute. I've been busy with other ventures and trying to stay happy and healthy. It hasn't worked very well. 

I also tried to buy a wharf, and in failing to do so interviewed more than 50 people working on or thinking about the Ocean. Across every conversation there was a common thread of maintaining cultural continuity in iconic trades, offset with earnest hope that the future is more collaborative. 

It was a year of migrations, shake-ups, business failures and successes. It is difficult to draw any sort of conclusion on how to prepare for that. But things will be different moving forward, and there is money back on the table. If we have learned anything, it is that you need infrastructure in place before shit hits the fan. Every technology we need for prosperous and accountable blue economy already exists. “All we need to do” is connect the dots. 

That's where you come in. The future is distributed, and the benefits available to large firms are possible for small and cooperative ventures through the use of technology. We live risk, and can use all the help we can get. We need investment in people, processes, and technology. That intentionally echos agile principles, because even though frameworks make me feel a little 🤮, language is like, important or something. 

I've said before that I want to align with one metric from the Maine State Strategic Plan: raise median wages. Also, increase economic production, but mostly the wages thing. You can do that the old extraction way, through ancient innovations like communication and stewardship. As long as we optimize extraction, things will trend toward overcapacity and consolidation. 

I was lucky to reminded of this yesterday by someone else in the industry who was frustrated by the ethical amalgam that is the “blue economy”. The health of the ocean never improves because we exploit it better. We are very lucky to still have some of our natural resources, but it's not anything the rush for a few billion dollars couldn't undo. 

We don't need 🌕 shots, we need new models of participation. If the answers were obvious and could be accomplished with time and money alone, they would be solved. Money helps of course. Some of my open questions from the pondering over past year include:

- Does trust matter if we have observability?
- If renewing and amending property rights were easier could terms be shorter?
- Can coordinated, incremental, organic growth be incentivized?

I wanted to buy Union Wharf because it could serve as a platform to accomplish the wages goal by empowering other people who needed quality marine facilities and services, and a way to build an accountable community. 

<Squalltalk center={[-70.2554061, 43.6539475]} zoom={16}/>

That's how it's been run so far (more later), but I figured you could sprinkle some technology on that Poole magic. I was disappointed to find out that the “SeaWork” trademark was taken. 

The property consists of warehouses, storage, docking and berthing, office, retail, restaurant, and parking income. It's truly amazing, and I'm interested to see what the Gulf of Maine Research Institute does with it. 

The main part is zoned Waterfront Central (WCZ), and abuts the Downtown Entertainment Overlay (DEOZ). The Portland zoning rules protect marine use, in a very useful way. There is a 180 day waiting period on leasing to non-marine uses, which can lead to low tenancy at times. The Proprietors of Union Wharf counteracted this by offering good deals to well behaved, anchor tenants. This is at the heart of why the property is so cherished, with greater that 90% occupancy. That means 10% could be available at any given time for short-term (startup) use. 

Across about 4 acres and 12 structures there are:

- 1,500+ linear feet of vessel berthing ($150)
- 250+ linear feet street frontage
- 60K+ square feet indoors ($17)
- 55K+ square feet yard and pier ($8)
- 19,230 square feet submerged leases through 2034
- 3K+ square feet wet and cold storage ($18)
- 30K+ square feet parking ($8)

Lease agreements bundle services and charge variable rates, which hides opportunity costs. Considered as a business, the property is worth $6M based on discounted cash flow for 10 years with modest growth. According to a marine construction expert, one should have $2.5M to deal with repairs and improvements over 5 years. So, the capitalization rate is 9-17%. 

The additional costs come in part from plans to [dredge the harbor to make CAD cells for storing hazardous sediments](https://www.portlandmaine.gov/1286/CAD-Cell-Planning). This will loosen pilings, maybe requiring simultaneous repair of the whole 1600+ linear feet of the wharf. 

Additional liability comes from possible dereliction of vessels. This has happened three times in Portland since 2000. In one notorious case the owner “sold” the boat to someone in the Old Port, only to drink the proceedings and go on to “sell” it again to someone else.

Filling berthing with working vessels prevents this, but fishing berthing is not a long term money maker. There is a strong lobster industry presence, and it is likely that from my conversations that they were one of the bids on the property. Close ties with the family and all that. Okay for steady month to month income, but maybe not a sustainable in the long term. Unfortunately, the deepest draft is about twenty feet, which prevent larger vessels for offshore wind support. You can put a decent research vessel there though, or a crew vessel. 

The previous owners are a waterfront fixture, and provided many fishermen their first berth. The job of a Wharfinger, like a bouncer, is to keep out bad tenants, even at loss of income. This dedication is evident in the common refrain I heard that Union Wharf is the ONLY property on commercial street worth investing in. 

So let's engage in a little make believe. You can see our $5K 3D architectural rendering below. Grey is hardscape and road, green is landscape, orange is existing structures to remain, yellow is new structure, and light blue is dedicated pier and marine berthing. 

<img src="/assets/union-wharf.png" alt="A Child Draws A Wharf Layout With Computers" style={{width: "512px"}}/>

(1) Mixed office, restaurant, and retail on Commercial Street. Currently, this is profitable, but holds mostly non-marine use and is a distraction from improving the utility of the property for marine industry. Sell it! But only after doubling the effective storefront space, and bringing safe pedestrian access 300 further southeast. A new connection to shared access road improves traffic flow. The profit can fund maintenance of the southwest side of the wharf, see (4). 

(2) Warehouse, offices, shared facilities. This is currently and oil spill response company, which I really hope will stay given the amount of large ships that come into the harbor. If they were to leave, it would be an excellent option for shipping and receiving and waste management.

(3) Marine campus, consisting of park, concourse, and private boat ramp. Design for runoff remediation and catchment, as well as periodic flooding with sea water. The is some consensus that most business on the water will be done with boats that fit on towed trailers, and having that capacity adjacent to storage and repair facilities would be a game changer for some operators. 

(4) Industrial condos and berthing. Marine use on flexible lower levels, with offices on upper level. That 180 day waiting period doesn't sound so bad when you have turn key facilities that new ventures can use short-term. I imagine something like this will happen with new owners. Includes 900-1000 feet of berth space for commercial vessels. Existing uses will consolidate over time into these facilities, which have dedicated water access and loading dock. All systems electrified to allow energy co-generation on south-facing roofs. The overall berthing capacity is high, and does not appear to pay for itself. You would need eliminate 500' feet of dedicated berthing, leaving 1000' contiguous. The lost 500' feet could still be available for transient vessels. Loss of working yard and parking spaces are resolved by adding “underground” parking and storage. This means building up. The waterfront code is good in this regard allowing 20-foot lower level, two 15-foot levels, and an additional 15 feet for HVAC. One can imagine a flexible industry condo, like this:
<img src="/assets/marine-condo.png" alt="Pixel Art Marine Light Industrial Condo"/>

(5) Port facility and maritime center. Elevation raised to accommodate underground parking and storage for tenants. Current maintenance yard will converted to a truck turn and loading dock with shared gantry cranes and electric fork trucks. Tie-ups for deep draft vessels is possible along the south and north-east edges. 

I'm going to be over here with my 🍿, seeing what happens. I'm a little disappointed not to be invited to the party. But you know, these folks have been working on their new ventures division for ten years, and brought together 30-35 parties in the deal. Maybe I'll be a tenant some day, or the Wharfinger. We do agree on one thing, that state action is insufficient to protect access to rights and equity. I just worry that venture capital and institution building can get in the way of doing real work. If all the money flows to organizations that do business development and market strategy rather than project execution, how do we get better terms for the human beings at the production end of that supply chain?

Leave these in the water and collect the reward or harvest the whole bed and sell it.


Oysters

Mussels are good! If they are too close to oysters though they will eventually take over.


Mussels

Sometimes if you are really nice and bring scientists seafood they will publish a paper revealing the true nature of things.  


Laboratory

Finfish are valuable but be careful bad things can happen if they escape. If the sea lice get to be too much you should talk to the scientists at the laboratory about collaborating on a paper.


Fish Pen

The Lighthouse has been around forever. It provides a bonus to nearby activities and protects boats from being lost at sea. 


Lighthouse

Gulls indicate that there is a hidden feature on a tile. These can be converted to activity tiles by being near a laboratory or being visited by a boat.


Herring Gull

Boat

Ocean tiles aren't empty. They are worth points but only next to other ocean tiles.   


Ocean

Land

Boats can put out and recover lobster traps! Don't molest yuor neighbors traps or you'll get in trouble. Karmicly and physically.


Lobster Buoys

Every good day starts at the dock! You'll have to fuel up here before you can go out and save the world. Gah but look at all that trash in water sure wish someone would take care of this place


Wharf

Your boats can turn into a diver to investigate underwater secrets or do harvest activities. It's not polite to run over divers that's why they fly such brightly colored flags.


Diver

Wind Turbine

Things break down on the water. If you're going to do business out there you better be ready to respond quickly.    


Damaged Wind Turbine

Ooey gooey sticky smelly mud. Full of tasty treats. Doesn't Jimmy's uncle have a fan boat or something?


Mud Flat

Looks like somebody messed up big time. 
