checkpoint

content/sundries/a-very-digital-artifact/index.md

@@ -69,12 +69,12 @@ described by three 3D points, where each point (but not necessarily each edge) o
 on the surface of the shape of the thing you're modeling. This format is popular with 3D printers,
 which is how I became familiar with it.
 
-STL is simple to implement and easy for a computer to read, but if you have a model in that
-format that you need to manipulate, like you want to merge it with another shape, you won't have a
-good time. In order to actually do things like change the shape of the model, it needs to be
-converted into a CAD program's native representation of a "solid body", which is pretty much what it
-sounds like: a shape made of a finite volume of "stuff", and NOT just an infinitesimally thin shell
-enclosing an empty volume, which is what the STL mesh is.
+STL is simple to implement and easy for a computer to read, but if you have a model in that format
+that you need to manipulate, like you want to merge it with another shape, you won't have a good
+time. In order to actually do things like that, it needs to be converted into a CAD program's native
+representation of a "solid body", which is pretty much what it sounds like: a shape made of a finite
+volume of "stuff", and NOT just an infinitesimally thin shell enclosing an empty volume, which is
+what a mesh is.
 
 In order for the CAD program to convert a mesh into a solid body, the mesh must be *manifold*,
 meaning, no missing faces (triangles), and with a clearly-defined interior and exterior (all
@@ -89,7 +89,7 @@ is the extension for a "STEP" file; STEP is supposed to be short for "standard f
 product data", so someone was playing pretty fast and loose with their initialisms, but I
 digress. The main thing about STEP files is that CAD programs can really easily convert them
 into their native internal solid body representation, which allows easy manipulation. Another thing
-about them is that a CAD program can usually turn an STL file into an STP file, unless the mesh is
+about them is that a CAD program can usually turn a manifold mesh into an STP file, unless the mesh is
 too complicated and your computer doesn't have enough RAM (*note: foreshadowing*[^chekhovs-ram]).
 
 ![an overly-complicated mesh of a cube][meshy-cube]
@@ -136,9 +136,9 @@ shabby!
 
 They provided the data in the form of [GeoTIFF](https://en.wikipedia.org/wiki/GeoTIFF)s, which are
 basically an image where each pixel represents one data point (so, a 30x30 square meter plot)
-centered at a particular location on the Earth's surface. It's a monochrome image, where height is
-mapped to brightness, so the lowest spot's value is `0` (black), and the highest spot is
-`65535`[^16-bit-ints] (brightest white).
+centered at a particular location on the Earth's surface. It's a monochrome image, where absolute
+height is mapped to absolute brightness of each pixel, and each pixel represents an exact location
+in the world.
 
 The only problem was that you could only download data covering up to 450,000 square kilometers at a
 time, so I had had to download a bunch of separate files and then mosaic them together. Luckily,
@@ -147,7 +147,7 @@ there's a whole suite of open source tools called
 `gdal_merge.py` (yes, the `.py` is part of the name of the tool that gets installed to your system
 when you install the GDAL tools), which does exactly what I wanted:
 
-> gdal_merge.py -o ca_topo.tif norcal_topo.tif centcal_topo.tif socal_topo.tif so_cent_cal_topo.tif norcal_topo_redux.tif last_bit.tif east_ca.tif
+> `gdal_merge.py -o ca_topo.tif norcal_topo.tif centcal_topo.tif socal_topo.tif so_cent_cal_topo.tif norcal_topo_redux.tif last_bit.tif east_ca.tif`
 
 This produced a file called `ca_topo.tif`. It was very large, in every sense:
 
@@ -194,28 +194,28 @@ Band 1 Block=40757x1 Type=Int16, ColorInterp=Gray
     Computed Min/Max=-130.000,4412.000
 ```
 
-If I may draw your attention to a couple things there, that's an image that's 40,757 pixels wide and
-35,418 pixels tall. The "pixel size" is 0.000277777777778 by 0.000277777777778; since each pixel is
-1 arc second, and 1 arc second is 1/3600th of a degree, and 1/3600 is 0.000277777777..., we
-can infer that the unit of size there is degrees of arc along the surface of the Earth[^wgs-ellipsoid], at a
-distance measured from the center of the planet. As previously mentioned, that translates into a size
-of roughly 30 meters. So if you were ever curious about how many 100-ish-foot squares you'd need to
-fill a rectangle that fully enclosed the entire border of California, then one billion,
-four-hundred-forty-three million, five-hundred-thirty-one thousand, and four-hundred-twenty-six
-(40,757 times 35,418) is pretty close.
+If I may draw your attention to a couple things there, the image is 40,757 pixels wide and 35,418
+pixels tall. The "pixel size" is 0.000277777777778 by 0.000277777777778; the units, given by the
+"angleunit", is degrees; 1 arc second is 1/3600th of a degree, which is 0.01754... They're degrees
+of arc along the surface of the Earth[^wgs-ellipsoid], at a distance measured from the center of the
+planet. As previously mentioned, that translates into a size of roughly 30 meters. So if you were
+ever curious about how many 100-ish-foot squares you'd need to fill a rectangle that fully enclosed
+the entire border of California, then one billion, four-hundred-forty-three million,
+five-hundred-thirty-one thousand, and four-hundred-twenty-six (40,757 times 35,418) is pretty close.
 
 The other units in there are under the "Coordinate System is" section, and are meters relative to
 the [World Geodetic System 1984](https://en.wikipedia.org/wiki/World_Geodetic_System) vertical
-datum; the very last line is the lowest and highest points in file, in meters from that WGS84
-baseline. If you were to view the file as though it were an image, it would look like this:
+datum; the very last line is the lowest and highest points in file, which are <a
+name="minmax-height"></a>-130 meters and 4,412 meters respectively, relative to the baseline height
+defined by the WGS84 ellipsoid. If you were to view the file as though it were an image, it would look like this:
 
 ![the ca_topo image; it's hard to make out details and very dark][small_ca_topo]
 *<center><sup><sub>if you squint, you can kinda see the mountains</sub></sup></center>*
 
-This is because the highest possible value an image like that could have for a pixel is 65,535, and
-the highest point in our dataset is only 4,412, which is not that much in comparison. Plus, it
-includes portions of not-California in the height data, and ideally, we want those places to be 0;
-we have a little more processing to do before we can use this.
+This is because the highest possible value an image like that could have for a pixel is
+65,535[^16-bit-ints], and the highest point in our dataset is only 4,412, which is not that much in
+comparison. Plus, it includes portions of not-California in the height data, and ideally, we want
+those places to be 0; we have a little more processing to do before we can use this.
 
 ## Cartography is complicated
 
@@ -247,14 +247,15 @@ is for reprojecting geotiffs. So all we have to do is take our 4326-projected ge
 `gdalwarp` to project it to 3857/Web Mercator, and then we can use the shapefile to mask off all
 other height data outside the border of California.
 
-It's almost *too* easy:
+It's almost *too* easy.
 
 > gdalwarp -t_srs EPSG:3857 ca_topo.tif ca_topo_mercator.tif
 
 This gives us a 3857-projected file called `ca_topo_mercator.tif`. It still has over a billion
-pixels in it and it's still almost the same size (though it's slightly different now, with the
-different projection); scaling it down is a last step, since at that point, it will no longer be a
-digital elevation map, it will just be an image. But we'll get there.
+pixels in it (it's a little bigger overall, but the aspect is
+much wider, with the different projection); scaling it down will be a very last step, since at that
+point, it will no longer be a digital elevation map, it will just be an image. We'll get there,
+just not yet.
 
 Cracking open `gdalinfo`, we get:
 
@@ -321,17 +322,18 @@ Upper Right (-12666831.611, 5178117.270) (113d47'17.10"W, 42d 6'51.50"N)
 Lower Right (-12666831.611, 3799580.326) (113d47'17.10"W, 32d16'33.21"N)
 Center      (-13296983.361, 4488848.798) (119d26'55.80"W, 37d21'21.69"N)
 Band 1 Block=36434x1 Type=Int16, ColorInterp=Gray
-
 ```
 
 You can see that the `PROJCRS[ID]` value is `"EPSG,3857"`, as expected. The "pixel size" is
-"34.591411...." since the "lengthunit" is "metre". But the number of pixels is different; it's not
-as wide, yet the coordinates of the bounding corners are the same as the original file (the latitude
-and longitude given as the second tuple).
+"34.591411...." and the "lengthunit" is "metre". But the number of pixels is different, and the
+shape is different, yet the coordinates of the bounding corners are the same as the original file's
+(the latitude and longitude given as the second tuple). This is all from the Web Mercator's different
+projection causing the aspect ratio to stretch horizontally, but it still represents the same area
+of the planet.
 
 ## The one custom script
 
-So, the next step was use our shapefile to mask out the California border in our geotiff. Here is
+So, the next step was use the shapefile to mask out the California border in the geotiff. Here is
 where GDAL failed me, and looking around now as I write this, I still can't find a specific GDAL
 tool for doing this. Given how useful I found all the other tools, I can't really complain, so I
 won't! It wasn't that hard to write something that would do it with other open source tools; I
@@ -384,11 +386,56 @@ I include that just in case anyone else ever needs to do this, and doesn't find
 of other examples out there already. This one is nice because you don't need to pre-process the
 shapefile into [GeoJSON](https://geojson.org/) or anything, the
 [Fiona](https://pypi.org/project/Fiona/1.4.2/) package handles things like that transparently for
-you, but don't think this is great python or something, it's the dumbest, quickest thing I crapped
-out to do the task I needed to be done.
+you, but don't think this is great Python or anything; it's the dumbest, quickest thing I could crap
+out to do the task I needed to be done[^the-real-treasure-is-the-gd-treasure].
+
+After running that script, I had a Web Mercator-projected geotiff that included data only for places
+inside the state border of California. It was still enormous; the mask didn't change the shape and
+you can't have non-rectangular images anyway, but at this point, I had the final desired
+dataset. It was time to turn it into a heightmap that we could use to make a mesh.
 
 ## A usable heightmap
 
+I've been trying to be careful about referring to the image file as a "dataset" or "geotiff", vs. a
+"heightmap". A geotiff file is not a regular image file, it includes particular metadata and data
+that is meant to be interpreted as a real map of the land; each pixel in it says something about an exact,
+actual location in the real world.
+
+A "heightmap" is an image file, like a geotiff, where each pixel's monochromatic intensity is meant
+to represent height above some lowest plane. The difference is that the height values are normalized
+so that the lowest value is 0, and the highest is the maximum possible value in the number
+range. For our geotiff, which uses 16-bit numbers as previously mentioned, that maximum possible
+value is 65,535. It also has no exact correspondence with anything else; it's not necessarily an
+accurate dataset, and won't include the GIS stuff like what projection it is, what the coordinate
+bounding boxes are, etc. But it *is* useful for turning into a mesh.
+
+And here I get to the [final GDAL tool](https://gdal.org/programs/gdal_translate.html) I used,
+`gdal_translate`. This is something that can read in a geotiff, and write out a different image
+format. When in doubt, [PNG](https://en.wikipedia.org/wiki/Portable_Network_Graphics) is fine, I
+always say. It's a simple format that nearly everything can read, and is compressed so it should be
+much smaller file.
+
+> `gdal_translate -of PNG -ot UInt16 -scale -130 4412 0 65535 masked_ca_topo.tif heightmap.png`
+
+Like we saw <a href="#minmax-height">earlier</a>, the lowest point we had was -130, and the highest
+was 4,412. The `-scale -130 4412 0 65535` arguments are saying, "anything with a height of -130
+should be set to 0, and anything with a height of 4,412 should be set to 65,535, and anything
+in-between should be set proportionately." This is a linear mapping, and preserves the relationships
+between vertical features (so, if something is twice as tall as another thing, that will
+still be true after being scaled), so it's "accurate" in a sense (*note: more foreshadowing*).
+
+Once I had the PNG file, I used the [ImageMagick](https://imagemagick.org/script/convert.php) `convert`
+command to scale the file down to a reasonable size. Finally, I had something I could use to make a
+mesh:
+
+![the heightmap made by doing a linear scale of height to brightness][scaled_heightmap]
+
+Pretty cool, right? I thought so! The detail is pretty great; that bright spot near the top is
+[Mt. Shasta](https://en.wikipedia.org/wiki/Mount_Shasta), for example;
+[Mt. Whitney](https://en.wikipedia.org/wiki/Mount_Whitney) is slightly taller, but not by much, and
+is part of a range so it doesn't stand out the way Shasta does. It was time to start making some 3D
+geometry with the heightmap!
+
 # A mesh is born
 
 # Test prints
@@ -424,6 +471,8 @@ given how much I had to blur and decimate)?
 
 [small_ca_topo]: small_ca_topo.png "a 'raw' heightmap of california and parts of nevada, arizona, and mexico"
 
+[scaled_heightmap]: scaled_heightmap.png "the heightmap made by doing a linear mapping of height to brightness"
+
 [^introspection]: The conclusion upon examination was, "I just wasn't thinking".
 
 [^math-computers]: I'm pretty sure this is more "represent shapes with math" than with a computer, but
@@ -445,3 +494,8 @@ so there you go.
 
 [^wgs-ellipsoid]: Technically, it's an arc along the WGS84 ellipsoid, which is a perfectly smooth
 *smushed* sphere, which more closely matches the real shape of the Earth vs. a perfectly round sphere.
+
+[^the-real-treasure-is-the-gd-treasure]: A friend posited at one point that my circuitous journey to
+the end product was the point, but I assured him that every step I took was trying to get to the end
+product as quickly and straightforwardly as possible. Still, I did in fact wind up learning a whole
+shitload of stuff, which is nice, I GUESS.