Get fully off of Google Maps geocoding API

[danvk]

After #174, I thought I was fully off of the Google Maps geocoding API, but it seems there are still a few uses.

With the geocache:

$ poetry run oldnyc/geocode/geocode.py --images_ndjson data/images.ndjson --output_format geojson --geocode > /tmp/images.geojson
...
-- Final stats --
   77 fifth
27080 title-cross
  600 title-address
 4060 gpt
   58 special
 1932 subjects
33807 (total)

vs. without:

$ rm -rf geocache
$ poetry run oldnyc/geocode/geocode.py --images_ndjson data/images.ndjson --output_format geojson --geocode > /tmp/images.geojson
... lots of warnings ...
-- Final stats --
   77 fifth
25380 title-cross
    0 title-address
 2865 gpt
   68 special
 2223 subjects
30613 (total)

[danvk]

• OSM is only used for intersections, not addresses.
• OSM geocoding is based on several CSV files containing street intersections.
• Modulo some normalizing of street names, I look for an exact match in those CSV files..
• If there is none, I might try to extrapolate streets to a point where they do interesct.

An example of a failure: 716380f

• Title is "Manhattan: Amsterdam Avenue - 193rd Street"
• This intersection does not exist, at least not today. 193rd Street starts and stops a bunch, but ends one avenue before Amsterdam.
• Google is able to geocode this to the right spot because Amsterdam/193rd is a defunct bus stop!

Another example: 716507f

• Title is "Manhattan: Battery Park - U.S.Custom House." which looks like an intersection but is not.
• Description is "Battery Park: A southeastward view, showing the State Street boundary, with the United States Custom House thereon. Photographers vantage point is the Whitehall Building at West Street."
• title-cross matches "Battery Park & U.S. Custom House". OSM & Google both take a crack at this and fail. This is where it hits the geocache.
• GPT extracts two place names and two intersections:
  • ✅ Battery Park
  • ✅ U. S. Custom House
  • ⚠️ State Street & Whitehall Street
  • ⚠️ West Street & Whitehall Street
• State Street & Whitehall Street geocodes successfully with OSM. This location is fine, though it's a happy coincidence that "Whitehall Street" works here. Locating to Battery Park POI would be better.

So maybe a lot of these geocache hits are failures?

[danvk]

Totally dropping Google:

 Before: 33,807
  After: 30,613

  Added: 0
Dropped: 3,194
Changed: 629

Dropping Google for intersections but not addresses:

 Before: 33,807
  After: 31,962

  Added: 0
Dropped: 1,845
Changed: 333

I reviewed 20 of the changes in the latter diff here:

• Drops: 9/10 losses, 1 small win
• Changes: 5 wins, 4 losses, 1 neutral

Generally speaking, Google is more aggressive about trying to snap to a match. Often that works, sometimes it doesn't.

A few specific cases where OSM could be improved:

• 726061f: OSM only has "Astoria Boulevard North" & "Astoria Boulevard South" whereas I think this needs to match just "Astoria Boulevard"
• 712960f: Vanderbilt Ave could be extrapolated to intersect 42nd (it stops at 43rd in OSM).
• 727463f: Google corrects a misspelling ("Newton" vs. "Newtown"). If GPT ran over this, OSM would get it right.
• 707234f: {"str1": "Ave. W", "str2": "Coney Island Ave.", "boro": "Brooklyn"} fails to geocode, but that intersection is in the CSV file, so it should really work.
• 719823f: I think OSM gets thrown off by "Fort Washington Aven" vs. Avenue

[danvk]

TL;DR: This change will be a little messy. Using OSM for addresses might be a good next step. There are a few bugs in the OSM geocoder that could be fixed.