nebkor/blog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

final polish

Browse source
This commit is contained in:
Joe Ardent 2023-01-19 22:34:47 -08:00
parent 16886e023c
commit 6a98cf409b
1 changed files with 101 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

191
content/sundries/a-thoroughly-digital-artifact/index.md
View file

@ -106,7 +106,7 @@ map](https://en.wikipedia.org/wiki/Digital_elevation_model) data, but I wound up
anything appropriate. Searching now with the wisdom of experience and hindsight, I found this, which
would have been perfect:
[https://apps.nationalmap.gov/downloader/](https://apps.nationalmap.gov/downloader/)
<https://apps.nationalmap.gov/downloader/>
Did I just accidentally miss it then? Did I find it and not recognize its utility because I didn't
know what I was doing *at all*? The world may never know, but at least now you can benefit from my
@ -209,17 +209,17 @@ the [World Geodetic System 1984](https://en.wikipedia.org/wiki/World_Geodetic_Sy
(distances from this reference surface in the dataset are within 16 meters of the true distance in
reality); the very last line is the lowest and highest points in file, which are <a
name="minmax-height"></a>-130 meters and 4,412 meters respectively, relative to the baseline height
defined by the WGS84 ellipsoid. If you were to view the file as though it were an image, it would
look like this:<a name="raw-dem"></a>
defined by the WGS84 ellipsoid. If you were to view the file with an image viewer, it would look
like this:<a name="raw-dem"></a>
![the ca_topo image; it's hard to make out details and very dark][small_ca_topo]
*<center><sup><sub>if you squint, you can kinda see the mountains</sub></sup></center>*
This is because the highest possible value an image like that could have for a pixel is
65,535[^16-bit-ints], and the highest point in our dataset is only 4,412, which is not that much in
comparison. Plus, it includes portions of not-California in the height data, and ideally, we want
those places to not be represented in our dataset; we have a little more processing to do before we
can use this.
It's almost completely black because the highest possible value an image like that could have for a
pixel is 65,535[^16-bit-ints], and the highest point in our dataset is only 4,412, which is not that
much in comparison. Plus, it includes portions of not-California in the height data, and ideally, we
want those places to not be represented in our dataset; we have a little more processing to do
before we can use this.
## Cartography is complicated
@ -337,11 +337,11 @@ of the planet.
## The one custom script
So, the next step was use the shapefile to mask out the California border in the geotiff. Here is
where GDAL failed me, and looking around now as I write this, I still can't find a specific GDAL
tool for doing this. Given how useful I found all the other tools, I can't really complain, so I
won't! It wasn't that hard to write something that would do it with other open source tools; I
didn't even bother checking this into a git repo or anything:
Now that we had our geotiff in the right projection, the next step was use the shapefile to mask out
the California border in it. Here is where GDAL failed me, and looking around now as I
write this, I still can't find a specific GDAL tool for doing it. Given how useful I found all the
other tools, I can't really complain, so I won't! It wasn't that hard to write something that would
do it with other open source tools; I didn't even bother checking this into a git repo or anything:
``` python
#!/usr/bin/env python3
@ -403,11 +403,12 @@ dataset. It was time to turn it into a heightmap that we could use to make a mes
I've been trying to be careful about referring to the image file as a "dataset" or "geotiff", vs. a
"heightmap". A geotiff file is not a regular image file, it includes particular metadata and data
that is meant to be interpreted as a real map of the land; each pixel in it says something about an exact,
actual location in the real world.
actual location in the real world. In our geotiff, a mountain would have to be more than twelve
miles high before it appeared as bright white[^zero-pixel-value].
A "heightmap" is an image file, like a geotiff, where each pixel's monochromatic intensity is meant
to represent height above some lowest plane. The difference is that the height values are normalized
so that the lowest value is 0, and the highest is the maximum possible value in the number
to represent height above some lowest plane. The difference is that the height values are *normalized*
so that the lowest height is 0, and the highest is the maximum possible value in the format's value
range. For geotiff digital elevation maps, which use 16-bit numbers as previously mentioned, that
maximum possible value is 65,535. But unlike a geotiff, a generic heightmap has no exact
correspondence with anything else; it's not necessarily an accurate dataset, and won't include the
@ -429,7 +430,7 @@ with a height of -130 should be totally dark, and anything with a height of 4,41
bright as possible, and anything in-between should be set proportionately." This is a linear
mapping, and preserves the relationships between vertical features (that is, if something is twice
as tall as another thing, that will still be true after being scaled), so in a sense, it's
"accurate" (*note: more foreshadowing*).
"accurate", but would it be good, was the question (*note: more foreshadowing*).
Once I had the PNG file, I used the [ImageMagick](https://imagemagick.org/script/convert.php) `convert`
command to resize the file down to a reasonable size. Finally, I had something I could use to make a
@ -450,7 +451,7 @@ assured me that [Blender](https://www.blender.org/), a free and open source 3D m
I'd dabbled with before, would work well. For example, here's a pretty high-level walk-through of
[how to use a heightmap to displace a mesh
plane](https://alanedwardes.com/blog/posts/create-meshes-from-height-maps-using-blender/), which is
definitely the first step I needed to take. Before too long, I had something that looked like this:
almost exactly what I first wanted to do. Before too long, I had something that looked like this:
![a very pointy california topo][pointy-california]
@ -458,31 +459,32 @@ At first glance, it looks OK, but there's so. much. detail. And it's very, very
looks jagged. Check out this close-up detail of Mt. Shasta:
![a very pointy mt shasta][pointy-shasta]
*<center><sup><sub>witch's hat-assed mountain</sub></sup></center>*
You can tell that would not be pleasant to touch, and being able to run your fingers along the shape
was a huge part of the appeal of the artifact.
You can tell it wounldn't be nice to touch, and being able to run your fingers along the shape
was a huge part of the appeal of having the physical object.
## Back to the realm of images
Given that it seemed like there were a couple semi-related problems with there being too much
detail, my first thought was to blur the heightmap, and then reduce the size of it. I used the
ImageMagick `convert` command [to blur the image](https://legacy.imagemagick.org/Usage/blur/) a
couple rounds, and then resized it down:
Given that it seemed like there were at least a couple semi-related problems from too much detail,
my first instinct was to blur the heightmap, and then reduce the size of it. I used the ImageMagick
`convert` command again [to blur the image](https://legacy.imagemagick.org/Usage/blur/) a couple
rounds, and then resized it down:
![first attempt at blurring the heightmap][blurry-linear-hm]
This was a little better, but still not great. A few more rounds of blurring and shrinking got me
A little better, but still not great. A few more rounds of blurring and shrinking got me
this:
![second round of blurring the heightmap][blurry-linear-hm-smaller]
With that version, I was able to produce some reasonably smooth-looking geometry in Blender:
With that version, I was able to produce some reasonable-looking geometry in Blender:
![a slightly smoother mesh][smoother-california-mesh]
Or so I thought.
As you can see, it's still very pointy. A lot of the high-frequency detail has been removed, which
It may have been smoother, it was still very pointy. A lot of the high-frequency detail has been removed, which
means it's not rough and jagged, but Shasta still looks ridiculous.
## A matter of scale
@ -493,37 +495,39 @@ required factors were so enormous that it distorted the geometry in an ugly way.
The State of California is very large, but for the sake of argument, let's pretend it's exactly 700
miles tall, from the southern tip to the northern border's latitude, going straight north; the real
length is close to that. Also for the sake of argument, let's say that the tallest mountain is 3
miles tall; the actual height is less than that, but that's OK. That means the ratio of height to
length is 3/700, or 0.0043-ish.
miles tall; the actual height is a little less than that, but that's OK, the argument holds more
strongly at lower height. That means the ratio of height to length is 3/700, or 0.0043-ish.
If you had a physically accurate topographic carving of California that was a foot long, the tallest
peak on the carving would be 0.0043 feet high, which is about 1/20th of an inch, or about 1.3
millimeters. You'd probably be able to tell with your fingers and even your eyes where Shasta was,
peak on the carving would be 0.0043 feet high, which is about 1/200th of an inch, or about 0.13
millimeters. You'd probably be able to tell with your fingers and maybe even your eyes where Shasta was,
and see that there was a faint line from the Sierra Nevadas, but that would be it. That's why it's
so hard to see the details in the <a href="#raw-dem">raw elevation data</a> geotiff.
In order to be able to see any detail, and to meet expectations about what a topographic carving is
supposed to look like, the height of the highest peaks needs to be exaggerated by something like
supposed to look like, the height of the highest peaks needs to be scaled up by something like
10-20x. My problem was that I was doing a linear scale; I was making *everything* 10-20x taller than
it "should" be, which was causing everything to look stretched and weird.
And even with that amount of exaggeration, some features were not showing up. For example, [this
2,000-foot tall mound in the Sacramento Valley](https://en.wikipedia.org/wiki/Sutter_Buttes), which
is faintly visible in the heightmap, is almost not there in the resulting mesh. It's about 1/7th the
height of Shasta, which is not all that much, when Shasta was represented by something 0.75 inches
tall.
And even with that amount of exaggeration, some low-elevation features were still not showing
up. For example, [Sutter Buttes, a 2,000-foot tall mound in the Sacramento
Valley](https://en.wikipedia.org/wiki/Sutter_Buttes), which is faintly visible in the heightmap, is
almost not there in the resulting mesh. It's about 1/7th the height of Shasta, which is not all that
much, when Shasta was represented by something 0.75 inches tall.
What I really needed was some non-linear way to scale the height, some way to exaggerate lower
altitudes more than higher ones. The highest points should stay as they were; they determine the
ultimate overall height, but as we saw, they overshadow lower features without a little help. An
easy way to do this is to take some fractional root (raise it to a power between 0.0 and 1.0) of the
linear scaling factor, and use that new value instead. For example, the graph of *x* raised to the
altitudes more than higher ones. The highest points should stay as high as they were; they determine
the ultimate overall height, but lower points should be given a relative boost. An easy way to do
this is to take some fractional root (raise a number to a power between 0.0 and 1.0) of the linear
scaling factor, and use that new value instead. For example, the graph of *x* raised to the
0.41th[^zero-forty-oneth] power looks like this:
![y = x^0.41 between 0 and 1][exp-plot]
Notice how values *at* 0 and 1 are the same as they would be with linear scaling, values *near* 0
rapidly get scaled upward, and by the time you get near 1, it looks almost linear again.
rapidly get scaled upward, and by the time you get near 1, it looks almost linear again. The linear
scaling function we'd initially used would just look like a straight line from the lower left corner
to the upper right.
Luckily, `gdal_translate` has an option to do this kind of scaling, so it was a quick
@ -539,24 +543,24 @@ which resulted in a mesh that looked something like this inside Blender:
![3D viewport in Blender showing a topo-displaced mesh that looks like
California][exp-scaled-blending]
Doesn't that look nicer? Notice how a bunch of things that were nearly invisible before, like that
mound near Sacramento, are easily visible. Check out the [Channel
Doesn't that look nicer? Notice how a bunch of things that were nearly invisible before, like Sutter
Buttes, are easily visible. Check out the [Channel
Islands](https://en.wikipedia.org/wiki/Channel_Islands_(California)) now plain as day! I was feeling
pretty good about having this whole thing wrapped up shortly, only a little late for the birthday.
# A dark period
# A dark age
What followed was two frustrating weeks attempting to get a manifold mesh out of Blender that was
small enough, by which I mean number of vertices and edges, so that
[FreeCAD](https://www.freecadweb.org/) could turn it into an STP file. Unfortunately, FreeCAD was
not a good tool for working with meshes, like creating them from a heightmap, so I had to use two
different tools.
[FreeCAD](https://www.freecadweb.org/) could turn it into an STP file. Unfortunately, FreeCAD is not
a good tool for doing fancy things with meshes, like creating them from a heightmap, so I had to use
two different tools.
This also meant that I would run into limits due to translation overhead when going between
them. Let me explain. I'd get a mesh in Blender, export it to a neutral mesh format like
This also meant that I would run into surprising limits when going between them. Let me explain. I'd
get a mesh in Blender, export it to a neutral mesh format like
[OBJ](https://en.wikipedia.org/wiki/Wavefront_.obj_file) that both programs understand well, and it
would be a 60 megabyte file. My computer has 32 **giga**bytes, more than 500 times more memory than
that, so you'd think it would not be a problem.
that, so you'd think it would not be a problem.
The act of asking FreeCAD to import that OBJ file as a *mesh*, and not even as a solid body, caused
the memory use to go to 21 gigabytes. This is a lot, but the computer still had plenty of room left
@ -564,11 +568,11 @@ in memory for things like "responding to the keyboard and mouse" or "redrawing t
screen". Everything at this point is still perfectly usable.
When I attempted to convert that mesh into a solid body, though, memory use ballooned up to
encompass all available RAM, and my system slowed to a nearly imperceptible crawl until my frantic
`ctrl-c`s were finally registered by the [signal
encompass all available RAM, and my system immediately came to a nearly imperceptible crawl until my
frantic `ctrl-c`s were finally registered by the [signal
handlers](https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html) in FreeCAD
before I could use it again. This happened *a lot*. At last, <a href="#prophecy">the prophecy</a> had
come to pass.
before I could use it again. This happened *a lot*. At last, <a href="#prophecy">the prophecy</a>
had come to pass.
I went through many rounds of attempting to clean up the mesh and reduce its complexity, but I don't
have many notes or intermediate artifacts from this time. A lot of that was being a beginner at both
@ -577,38 +581,41 @@ not knowing how to do a particular thing inside each program. A lot was inexperi
I did not know how much detail was essential, and I did not have a lot of experience with digital
modeling in the first place. The workflow was very manual, and cycles were fairly long, which made
it hard to try a bunch of things in quick succession as experiments. All those things and more
conspired to make this portion of the process a total slog with very little to show of.
conspired to make this portion of the process a total slog with very little to show off.
# Test prints
Finally, after a couple weeks of trying and failing to get something into FreeCAD that I could then
work with (like melding it with a thick base and trimming that base to follow the shape of the
state), I had had enough. I have some notes from the time, from right after I'd started a test print
of my latest model:
Eventually, after a couple weeks of trying and failing to get something into FreeCAD that I could then
work with (like merging it with a thick base and trimming that base to follow the shape of the
state), I had had enough. I was just going to send the shop an STL file and forget about trying to
get an STP file. I have some notes from then, right after I'd started my first test print:
> I'm finally printing something out. I've given up on converting it into [something CAD-friendly];
it seems this is a Hard Problem, but I'm not sure why. My goal with doing that was to give a
CAD-friendly file to a local CNC milling shop, per their request, since when I suggested a
mesh-based file (STL), the guy was like "I can't do much manipulation with that to make it more
manufacturable, so a real CAD file would be best".
> it seems this is a Hard Problem, but I'm not sure why. My goal with doing that was to give a
> CAD-friendly file to a local CNC milling shop, per their request, since when I suggested a
> mesh-based file (STL), the guy was like "I can't do much manipulation with that to make it more
> manufacturable, so a real CAD file would be best".
>
> But at least with an STL file, I can print it myself. So that's going now, we'll see how it turns
out in no less than eight hours.
> out in no less than eight hours.
>
> I haven't really done anything else with my computer besides this for a while.
When that print was done, here's what it looked like:
![a piece of literal dogshit][crappy_test_print]
*<center><sup><sub>don't look at me, I'm hideous</sub></sup></center>*
In case you were not revolted enough, then please allow me to direct your gaze upon this eldritch abomination:
In case you were not revolted enough, then please allow me to direct your gaze toward this eldritch
abomination:
![close-up of extremely bad print results][crappy-close-up]
*<center><sup><sub>what did I just say about looking at me</sub></sup></center>*
As bad as it looked, it felt even worse to touch. Setting aside the hideous base with its weird
artifacts due to those areas not being a single flat polygon, but rather several polygons that were
not parallel or co-planar (modeling artifact), there was still just too much high-frequency detail in the
terrain, and it was a total mismatch with the 3D printed medium.
visual artifacts due to those areas not being a single flat polygon, but rather several polygons
that were not parallel, there was still just too much high-frequency detail in the terrain, and it
was a total mismatch with the 3D printed medium.
The real thing was going to be carved out of wood by a [CNC
mill](https://all3dp.com/2/what-is-cnc-milling-simply-explained/), which uses a drill-like component
@ -619,7 +626,7 @@ not only ugly, it was also completely unnecessary.
## Just try harder
I was eager to get something into the CNC shop's hands at this point, but I also knew that this
I was very eager to get something into the CNC shop's hands at this point, but I also knew that this
model was not acceptable. So, I resolved to brutally simplify the geometry until I got something
that was workable inside FreeCAD.
@ -644,17 +651,16 @@ Check out this close-up of Mt Shasta:
![close-up of Shasta in the final model][final-shasta]
*<center><sup><sub>a chonkier, less lonesome Mt Shasta</sub></sup></center>*
Present, but not obnoxious. I printed out another test print to make sure it looked as good in
Present, but not obnoxious. I printed out a second test print to make sure it looked as good in
physical reality:
![the final test print of the final model][final-print]
Verdict: yes. If you want, you can visit
[https://www.printables.com/model/240867-topographic-california](https://www.printables.com/model/240867-topographic-california)
and download the 3D printer file to print it yourself at home. If you don't have a 3D printer, you
can still look at and interact with a 3D model of it in the browser on that site, so it's still kind
of neat. A couple different strangers uploaded pictures of their prints of it, which I thought was
cool!
<https://www.printables.com/model/240867-topographic-california> and download the 3D printer file to
print it yourself at home. If you don't have a 3D printer, you can still look at and interact with a
3D model of it in the browser on that site, so it's still kind of neat. A couple different strangers
uploaded pictures of their prints of it, which I thought was cool!
I brought the mesh into FreeCAD and finally was able to create the STP[^fancy-iges] file the shop had
asked for, a mere twenty-five days after I'd last spoken with them.
@ -678,14 +684,14 @@ The shop came back with,
>
> Let me know if this will work for you or not. If you think it will, I will try to program the toolpaths and see what it will look like.
I definitely didn't want to lose the sharp seams in the bottoms of the valleys.
I definitely didn't want to lose the sharp seams in the bottoms of the valleys!
> Me: I guess what I was really saying is that if some detail is lost due to using a larger cutting
head that's probably fine. I wouldn't necessarily want the valleys to be made more concave than they
already are, though. Does that make sense?
> head that's probably fine. I wouldn't necessarily want the valleys to be made more concave than
> they already are, though. Does that make sense?
>
> Shop: Yes, that makes sense. I can use a Vee cutter and it will cut the sharp edges in the
valleys."
> valleys."
It felt nice to be understood! Next came the issue of cost:
@ -705,9 +711,9 @@ One of the things that my wife had said she wanted to do with the carving of Cal
finish it herself, so the coarser 0.01-inch step-over cut was not really a problem. Even the
0.005-inch cut would still require a final sanding before staining or sealing.
The "larger one" the shop referred to was for a 20-inch wide carving, which would be huge anyway; 12
inches was fine. Still, $350 was at the top of what I had hoped/expected to spend. I hoped it was
worth it!
The "larger one" the shop referred to was for a 20-inch wide carving, which would be way too huge
anyway; 12 inches was fine. Still, $350 was at the top of what I had hoped/expected to spend. I
hoped it was worth it!
After a few more back-and-forths and days, I got a message from the shop saying it was ready. They
also said,
@ -745,13 +751,13 @@ MISSION ACCOMPLISHED, HAPPY *<sub>belated</sub>* BIRTHDAY!
# Thank yous
Obviously, I have tons of people to thank for their help with this, either directly or
indirectly. First and foremost, my wife, for everything but especially the inspiration and also
indirectly. First and foremost, my wife, for everything, but especially for the inspiration and also
patience with me during this process.
A close second goes to Steve at [Triumph CNC](https://www.triumphcnc.com/). He asked me what I was
going to do with it, and when I said give it to my wife as a gift, he said, "Oh, that's great! I
feel even better about using the smaller step-over now." If you need some CNC milling done in Los
Angeles, maybe give them a call!
A close second for this project goes to Steve at [Triumph CNC](https://www.triumphcnc.com/). He
asked me what I was going to do with it, and when I said give it to my wife as a gift, he said, "Oh,
that's great! I feel even better about using the smaller step-over now." If you need some CNC
milling done in Los Angeles, maybe give them a call!
Along the way during this journey I got a lot of feedback and suggestions from friends and
colleagues, so thank you, 'rades[^short-for-comrades]!
@ -760,8 +766,8 @@ Of course, this would all have been unthinkably difficult not so long ago, but t
NASA's missions and public GIS datasets, almost anyone can do something like this.
And not just public, government data and organizations, but private, passion-driven free software
projects like Blender and FreeCAD rival functionality available only in multi-thousand-dollar
commercial packages. I'm in awe of their accomplishments; they are true wonders of the modern world.
projects like Blender and FreeCAD that rival functionality found in multi-thousand-dollar commercial
packages. I'm in awe of their accomplishments; they are true wonders of the modern world.
# Things I learned, and some lessons
@ -875,6 +881,11 @@ the end product was the point, but I assured him that every step I took was tryi
product as quickly and straightforwardly as possible. Still, I did in fact wind up learning a whole
shitload of stuff, which is nice, I GUESS.
[^zero-pixel-value]: I'm not actually sure what the "0" altitude pixel value is. It can't actually
be 0, because the numbers in the file can't be negative, and there are deep valleys on the earth's
surface. But it's clearly not that high a value, otherwise, when you viewed the geotiff as an image,
it would be closer to white or gray than black.
[^time-to-mesh]: Based on the timestamps of the files in the directory where I was working on this
project, it took about ten days from the time I first downloaded a geotiff dataset to having the
heightmap shown above, so you can imagine all the dead-ends I went down and did not share in this