Oceanicsdotio

Dear NOAA

December 15, 2019


This is the body of a public comment submitted to NOAA in response to their draft artificial intelligence strategy

It is difficult to convey the mixture of excitement and disappointment at reading the draft artificial intelligence strategy. Excitement in spirit, disappointment in body. Given widespread evidence of successful adoption, you determined AI low-risk, and of national importance. Your goals in the service of efficiency, effectiveness, and coordination are in command-and-control terminology, and implicitly structure execution and assessment of the strategy. I would like to provide this public comment, as an oceanographer and software engineer working on AI applications in service of prosperity, agency, and cooperation.

Structure—NOAA will create a Center for AI (NCEI) and an AI technical committee. These entities will prioritize related budget lines, develop an “open source” science gateway, adopt cloud-native computing, organize events, and have executive-level conversations. The outcome and performance indicator is ”[exponential] increase in use and utility across all of NOAA”.

I question the need for a permanent center. Administratively or geographically concentrating domain expertise works against boundary spanning, and excludes participation. The performance metric here manages to be both vague and over-optimistic, simultaneously consolidating authority, mandating action across all offices, and pushing those services back to the public. Considering there is a major current of concern over ethics in AI, the committee or other bodies should have as much transparency and inclusion as possible.

An open source gateway is great, but exposing the inner workings of the data and service access layers doesn’t do anything for observability and interoperability of models and services. You end up with arbitrary data format requirements getting baked into RFAs (EPA, WaterML). The more industry functions and standards that NOAA assumes as canon, the less incentive there is for innovation and competition in industry.

Innovate—NOAA will innovate by institutionalizing AI and adopting a “requirements-based” process. This process will be continuously updated to keep pace with technology. Funding priorities will be communicated through FFO and RFPs, and prize competitions. Adoption will be tracked using the internal NRDD, and research projects at partner institutes evaluated with yet-to-be developed metrics. Promising methods will be evaluated through multiple “testbeds”.

Innovation is not meticulous curation and project management, nor is it sourcing your next idea through predatory practices like competitions. Innovation is giving creative people resources to synthesize new ideas, and getting out of the way. It is not the job of the government to pick winners and losers. Your process will not keep pace with technology, and adding more steps in the development pipeline through performance monitoring and gate-keeping will undo gains from automation. Internal tracking in NRDD with annual public reports is not an improvement, and doesn’t contribute to the open community of practice that has inspired this strategy.

The most innovative thing, would be to proactively seeking out those working in the space you propose to occupy, and enable constructive progress toward analytics products with net social and environmental gains. To date, NOAA has limited experience with artificial intelligence, exploring applications in robotic survey, data quality, and image analysis. It is distressing that the applications cited are image-based analysis tasks using decades-old methods, while ripe areas of human-machine interaction, graph reasoning, risk analysis, and behavior are unacknowledged.

Operationalize—NOAA will “rapidly accelerate” from basic research to algorithmic products, by taking things that already exist in industry and academia, and developing internal technical documentation that will be updated once per year. They will then evaluate these adopted technologies for re-commercialization, if they pass the pre-operational test phase. Annual reports will be used to build awareness and evaluate investigator performance.

What is the jounce on that rapid acceleration, and how you are going to get there by only assessing the state of knowledge once per year? We are in the middle of unprecedented environmental crises, so I agree that speed it essential. Live a little, distribute risk. Fund many projects which seem likely to fail. Fund identical projects in different places. Get people talking and co-operating on issues, provide a non-prescriptive forum for that. State clearly your needs and how much you are willing to pay, and encourage federated solutions that enable small companies and labs to compete with vertically integrated contractors and service providers.

Partner—NOAA will expand partnerships in AI, with a focus on the academic/research communities, interagency cooperative agreements, and public-private agreements through CRADA and SBIR. They will contribute to NSTC committee on AI, and engage in conference and workshop circuit.

Using existing approaches will result in status quo results. To go faster we need new funding and partnership mechanisms that enable small high-risk research projects by diverse participants, and incentivizes the validation and replication of work that goes into training and parameterizing models. We need methods to crowd-source knowledge, and rapidly disseminate it. Be a facilitator, not a domain expert. Sending executives to conferences is not a national strategy.

Train—NOAA will provide their staff with tailored training, using existing programs, partner guidance, and MOOCs and such. Staff will be rotated through offices to enhance cross-pollination, and positions and performance plans will be updated. They will support and lead collaborative conference and workshops, and “actively encourage” grad and co-op training to improve recruitment.

Notably absent is a focus on training a general workforce, discussion of diversity or inclusion, much less training in how AI and ML can contribute to inequity. There also seems to be an emphasis on optimizing people, without much attention given to the effects of moving people or altering their positions. People are understandably uncertain of their professions.

The decentralized use of AI resulted in redundancy and limited scope, but those successes were through the agency of individuals. Data literacy is crucial in the near-term, but often the way people come into it is through problem -> solution logic. Encourage the practices and people that arose organically. Making it a supply problem before there is a demand is premature optimization. It’s hard to get people because your recruiting pipeline sucks, not because there is a shortage of people who can learn on the job.