checkpoint post, almost done

content/sundries/a-very-digital-artifact/index.md

@@ -209,7 +209,7 @@ the [World Geodetic System 1984](https://en.wikipedia.org/wiki/World_Geodetic_Sy
 reality); the very last line is the lowest and highest points in file, which are <a
 name="minmax-height"></a>-130 meters and 4,412 meters respectively, relative to the baseline height
 defined by the WGS84 ellipsoid. If you were to view the file as though it were an image, it would
-look like this:
+look like this:<a name="raw-dem"></a>
 
 ![the ca_topo image; it's hard to make out details and very dark][small_ca_topo]
 *<center><sup><sub>if you squint, you can kinda see the mountains</sub></sup></center>*
@@ -477,8 +477,181 @@ this:
 
 With that version, I was able to produce some reasonably smooth-looking geometry in Blender:
 
+![a slightly smoother mesh][smoother-california-mesh]
+
+Or so I thought.
+
+As you can see, it's still very pointy. A lot of the high-frequency detail has been removed, which
+means it's not rough and jagged, but Shasta still looks ridiculous.
+
+## A matter of scale
+
+The problem was that I was doing a linear scaling of the height of features in the data, and the
+required factors were so enormous that it distorted the geometry in an ugly way.
+
+The State of California is very large, but for the sake of argument, let's pretend it's exactly 700
+miles tall, from the southern tip to the northern border's latitude, going straight north; the real
+length is close to that. Also for the sake of argument, let's say that the tallest mountain is 3
+miles tall; the actual height is less than that, but that's OK. That means the ratio of height to
+length is 3/700, or 0.0043-ish.
+
+If you had a physically accurate topographic carving of California that was a foot long, the tallest
+peak on the carving would be 0.0043 feet high, which is about 1/20th of an inch, or about 1.3
+millimeters. You'd probably be able to tell with your fingers and even your eyes where Shasta was,
+and see that there was a faint line from the Sierra Nevadas, but that would be it. That's why it's
+so hard to see the details in the <a href="#raw-dem">raw elevation data</a> geotiff.
+
+In order to be able to see any detail, and to meet expectations about what a topographic carving is
+supposed to look like, the height of the highest peaks needs to be exaggerated by something like
+10-20x. My problem was that I was doing a linear scale; I was making *everything* 10-20x taller than
+it "should" be, which was causing everything to look stretched and weird.
+
+And even with that amount of exaggeration, some features were not showing up. For example, [this
+2,000-foot tall mound in the Sacramento Valley](https://en.wikipedia.org/wiki/Sutter_Buttes), which
+is faintly visible in the heightmap, is almost not there in the resulting mesh. It's about 1/7th the
+height of Shasta, which is not all that much, when Shasta was represented by something 0.75 inches
+tall.
+
+What I really needed was some non-linear way to scale the height, some way to exaggerate lower
+altitudes more than higher ones. The highest points should stay as they were; they determine the
+ultimate overall height, but as we saw, they overshadow lower features without a little help. An
+easy way to do this is to take some fractional root (raise it to a power between 0.0 and 1.0) of the
+linear scaling factor, and use that new value instead. For example, the graph of *x* raised to the
+0.41th[^zero-forty-oneth] power looks like this:
+
+![y = x^0.41 between 0 and 1][exp-plot]
+
+Notice how values *at* 0 and 1 are the same as they would be with linear scaling, values *near* 0
+rapidly get scaled upward, and by the time you get near 1, it looks almost linear again.
+
+Luckily, `gdal_translate` has an option to do this kind of scaling, so it was a quick
+
+> `gdal_translate -of PNG -ot UInt16 -scale -130 4412 0 65535 -exponent 0.41 ca_topo.tif
+exponentially_scaled_heightmap.png`
+
+and a couple rounds of blurring, and I had the following heightmap:
+
+![a non-linearly scaled heightmap][lo-rez_exp_blurred]
+
+which resulted in a mesh that looked something like this inside Blender:
+
+![3D viewport in Blender showing a topo-displaced mesh that looks like
+California][exp-scaled-blending]
+
+Doesn't that look nicer? Notice how a bunch of things that were nearly invisible before, like that
+mound near Sacramento, are easily visible. Check out the [Channel
+Islands](https://en.wikipedia.org/wiki/Channel_Islands_(California)) now plain as day! I was feeling
+pretty good about having this whole thing wrapped up shortly, only a little late for the birthday.
+
+# A dark period
+
+What followed was two frustrating weeks attempting to get a manifold mesh out of Blender that was
+small enough, by which I mean number of vertices and edges, so that
+[FreeCAD](https://www.freecadweb.org/) could turn it into an STP file. Unfortunately, FreeCAD was
+not a good tool for working with meshes, like creating them from a heightmap, so I had to use two
+different tools.
+
+This also meant that I would run into limits due to translation overhead when going between
+them. Let me explain. I'd get a mesh in Blender, export it to a neutral mesh format like
+[OBJ](https://en.wikipedia.org/wiki/Wavefront_.obj_file) that both programs understand well, and it
+would be a 60 megabyte file. My computer has 32 **giga**bytes, more 500 times more memory than
+that, so you'd think it would not be a problem.
+
+The act of asking FreeCAD to import that OBJ file as a *mesh*, and not even as a solid body, caused
+the memory use to go to 21 gigabytes. This is a lot, but the computer still had plenty of room left
+in memory for things like "responding to the keyboard and mouse" or "redrawing the
+screen". Everything at this point is still perfectly usable.
+
+When I attempted to convert that mesh into a solid body, though, memory use ballooned up to
+encompass all available RAM, and my system slowed to a nearly imperceptible crawl until my frantic
+`ctrl-c`s were finally registered by the [signal
+handlers](https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html) in FreeCAD
+before I could use it again. This happened a lot.
+
+I went through many rounds of attempting to clean up the mesh and reduce its complexity, but I don't
+have many notes or intermediate artifacts from this time. A lot of that was being a beginner at both
+Blender **and** FreeCAD, though there's so much educational material that I was rarely held back by
+not knowing how to do a particular thing inside each program. A lot was inexperience in the domain;
+I did not know how much detail was essential, and I did not have a lot of experience with digital
+modeling in the first place. The workflow was very manual, and cycles were fairly long, which made
+it hard to try a bunch of things in quick succession as experiments. All those things and more
+conspired to make this portion of the process a total slog with very little to show of.
+
 # Test prints
 
+Finally, after a couple weeks of trying and failing to get something into FreeCAD that I could then
+work with (like melding it with a thick base and trimming that base to follow the shape of the
+state), I had had enough. I have some notes from the time:
+
+> I'm finally printing something out. I've given up on converting it into a parametric
+CAD-like object; it seems this is a Hard Problem, but I'm not sure why. My goal with doing that was
+to give a parametric CAD file to a local CNC milling shop, per their request, since when I suggested
+a mesh-based file (STL), the guy was like "I can't do much manipulation with that to make it more
+manufacturable, so a real CAD file would be best".
+>
+> But at least with an STL file, I can print it myself. So that's going now, we'll see how it turns
+out in no less than eight hours.
+>
+> I haven't really done anything else with my computer besides this for a while.
+
+When that print was done, here's what it looked like:
+
+![a piece of literal dogshit][crappy_test_print]
+
+In case you were not revolted enough, I invite you to cast your gaze upon this eldritch abomination:
+
+![close-up of extremely bad print results][crappy-close-up]
+
+As bad as it looked, it felt even worse to touch. Setting aside the hideous base with its weird
+artifacts due to those areas not being a single flat polygon, but rather several polygons that were
+not parallel or co-planar (modeling artifact), there was just too much high-frequency detail in the
+terrain, and it was a total mismatch with the 3D printed medium.
+
+The real thing was going to be carved out of wood by a [CNC
+mill](https://all3dp.com/2/what-is-cnc-milling-simply-explained/), which uses a drill-like component
+to carve away pieces of the material you're working with. This means that there's a tiny spinning
+bit with a definite, finite size, and any detail in the model smaller than the end of that spinning
+bit would likely be impossible to carve with it. This meant that all that high-frequency detail was
+not only ugly, it was also completely unnecessary.
+
+## Just try harder
+
+I was eager to get something into the CNC shop's hands at this point, but I also knew that this
+model was not acceptable. So, I resolved to brutally simplify the geometry until I got something
+that was workable inside FreeCAD.
+
+First off, I made the heightmap even smaller, only 500 pixels wide. Fewer pixels means fewer details
+for turning into a displacement map for a mesh! I also removed the Channel Islands from the
+heightmap, resulting in this final mesh displacement input:
+
+![it's the final heightmap][final-heightmap]
+*<center><sup><sub>it's the final heightmap (doot-doot-doot-doot,
+doot-doot-doot-doot-doot)</sub></sup></center>*
+
+Inside Blender, I'd gotten quite proficient at running through the steps to generate a mesh from a
+heightmap, and once I'd done that, I went through several rounds of [mesh
+simplification](https://graphics.stanford.edu/courses/cs468-10-fall/LectureSlides/08_Simplification.pdf);
+the geometry was practically homeopathic.
+
+![the final model in blender][final-model]
+
+Check out this close-up of Mt Shasta:
+
+![close-up of Shasta in the final model][final-shasta]
+*<center><sup><sub>a less lonely Mt Shasta</sub></sup></center>*
+
+Present, but not obnoxious. I printed out another test print to make sure it looked as good in
+physical reality:
+
+![the final test print of the final model][final-print]
+
+Verdict: yes. If you want, you can visit
+[https://www.printables.com/model/240867-topographic-california](https://www.printables.com/model/240867-topographic-california)
+and download the 3D printer file to print it yourself at home. If you don't have a 3D printer, you
+can still look at and interact with a 3D model of it in the browser on that site, so it's still kind
+of neat. A couple different strangers uploaded pictures of their prints of it, which I thought was
+cool!
+
 # Final cut
 
 # Thank yous, lessons learned, and open questions
@@ -498,7 +671,17 @@ given how much I had to blur and decimate)?
 
 ---
 
-[main_image]: PXL_20220723_214758454.jpg "A plywood slab carved with CNC into a topographic representation of California"
+[final-print]: final_printed.jpg "the final test print of the final model"
+
+[final-shasta]: final_shasta.png "close-up of Shasta in the final model"
+
+[final-model]: final_ca_topo_blend.png "the final model in Blender"
+
+[final-heightmap]: final_heightmap.png "it's the final heightmap (sick synthesizer riff blasts)"
+
+[crappy-close-up]: crappy_test_print_close_up.jpg "close-up of extremely bad print results"
+
+[main_image]: wood_ca_on_table.jpg "A plywood slab carved with CNC into a topographic representation of California"
 
 [programmers_creed]: /images/programmers_creed.jpg "jfk overlaid with the programmer's creed: we do these things not because they are easy, but because we thought they were going to be easy"
 
@@ -518,6 +701,16 @@ given how much I had to blur and decimate)?
 
 [blurry-linear-hm-smaller]: lo-rez_blurred_hm3.png "second round of blurring the heightmap"
 
+[smoother-california-mesh]: blending_california.png "slightly smoother mesh in blender"
+
+[exp-plot]: exponential_plot.png "a graph of the function `y = x^0.41` between 0 and 1"
+
+[lo-rez_exp_blurred]: lo-rez_exp_blurred.png "nearly final heightmap, using exponential scaling to exaggerate lower altitudes"
+
+[exp-scaled-blending]: non-linear_scaling_of_ca_height_data.png "You can see how Shasta doesn't stick out so much when the other hills are brought up a bit relatively speaking"
+
+[crappy_test_print]: ca_topo_crappy_test_print.png "a piece of literal dogshit"
+
 [^introspection]: The conclusion upon examination was, "I just wasn't thinking".
 
 [^math-computers]: I'm pretty sure this is more "represent shapes with math" than with a computer, but
@@ -550,3 +743,4 @@ project, it took about ten days from the time I first downloaded a geotiff datas
 heightmap shown above, so you can imagine all the dead-ends I went down and did not share in this
 write-up.
 
+[^zero-forty-oneth]: I think this was just the first fractional value that I tried, and it was fine.

content/sundries/a-very-digital-artifact/plot.gnuplot

@@ -0,0 +1,9 @@
+f(x) = x**0.41
+set terminal png size 600,600
+set output 'exp_scaling.png'
+set grid
+
+set xrange [0:1]
+set yrange [0:1]
+
+plot f(x) lw 3 notitle