Andrej's website and blog

Ansible: pure (only in its) pragmatism

Unknown — Tue, 15 Apr 2025 00:00:00 +0000

Opinionated and outspoken as we programmers are, I feel that Ansible gets an outsized amount of hate, not all of which is warranted </aside>

I’ve been using Ansible for the better part of a decade. I think I can provide some commentary on its merits and shortcomings. More importantly, I’ll try to analyze the approach it takes, its stature in the community and the implications of both to its utility.

What it is, what it does</h2>
It’s a tool that allows you to remotely (and locally) do various administration tasks, to put remote systems into some state, with an accent on idempotence.
The main course is the ability to get things to happen on remote machines via SSH or WinRM (Windows’ analogue) in an idempotent way. You declare the desired state, for example: a folder should exist at some path, with such-and-such permissions and ownership. If it doesn’t - Ansible creates it. If it does, but the permissions are wrong, it changes the permissions. If all the required conditions are already met, nothing happens.
The same is available for all the ordinary administration tasks you’d expect, like managing users, groups, `systemd</code> units, firewall configuration, packages in the package manager, etc. It also gives you generic file editing capabilities (altering lines or blocks), which integrates the Jinja2</a> templating for files and vars.`
Ansible’s lineinfile</code> and blockinfile</code> iteratively manage comment-delimited sections of files, or single lines identified by regex expressions, as they’re encountered in the Ansible code. This is unlike Puppet</a>, which would collect data and template it into a file in a single place. </aside> You can also imperatively run an executable or a piece of shell code, but that is discouraged, and if it’s unavoidable, you’re encouraged to wrap it in a way that makes it somewhat idempotent (check if something is in the desired state, only alter it if it isn’t). Ansible also adds a layer of universal control logic, with looping, conditions, function-like code reuse (roles), try-catch equivalents, inclusion, et cetera. This is all available through a YAML-based DSL akin to Kubernetes or Docker Compose, but with Jinja2 templating available everywhere. Ansible has by far the biggest and most prolific community of all the tools in the category, and by far the largest base of plugins and reusable code (roles) available through its package manager - repository combo known as Ansible Galaxy</a>. Apart from things you can have it do remotely on another machine, there’s a large number of official and community based integrations with APIs, so you may easily obtain credentials from HashiCorp Vault, register runners in GitLab, import and export Grafana dashboards, etc. Design choices</h2> First obvious thing is that Ansible relies much on file hierarchy, file naming, and files in general. Roles are especially heavy on subfolders, files and hiararchy</a>. </aside> As such, you’re expected to organize your playbooks (scripts you run directly) in a certain way, your inventory (hosts and groups) in another way, and roles (callable, reusable code, a cross between functions and modules), in a third way. It’s not hard to grasp but isn’t exactly intuitive either. You’ll be consulting the documentation a lot. Ansible doesn’t have its own language, but a style or DSL of YAML that’s relatively consistent. I see this being sneered at, but it has it’s upsides, too: familiarity, a decent solution for multi-line strings, availability of anchors (though those are not frequently used because Ansible has other mechanisms), support for parsing in all major languages if you want to generate, check or modify the code programmatically. Ansible was always marketed as having simplicity™ as one of its goals. You might not like its brand of simplicity: only global variables; worse, dynamically derived ones get mangled to be shared per host during a run, but are thankfully accessible in the global namespace</li> no variable namespacing, prefixes being the preferred way</li> no arguments, all args are global variables, you’re just supposed to prefix them with the role name</a>; all you C programmers put your hands up in the air</li> due to the simple model of code reuse, there are include</a> and import</a> with different semantics</li> </ul> Other obvious downsides include: role argument documentation and validation require duplication</a>, especially for defaults (you have to specify it both in the meta</code> and in defaults</code>), and are relatively recent</li> role-task distinction not that sensible, roles could have been structured more efficiently</li> the insistence on specifying both the inventory and the hosts in all cases</li> </ul> The role of roles has de facto changed over time - initially it was meant to be “set of tasks and helpers that enable a server to fulfill some role” to “a more complex piece of reusable code that you may call from a different place”. I’d say that the only thing pure about Ansible is its pragmatism </aside> Ansible is not a purely declarative tool by any means. Even though the single tasks are meant to be declarative to facilitate idempotence, the order of them matters, not to mention that you can run commands imperatively. In that way, it’s really not unlike running some shell scripts via SSH, just with some lipstick. Vs Shell scripts</h2> Your company is unlikely to be in the business of writing pretty deployment shell scripts </aside> It might cross your mind that you could do everything Ansible does with SSH and shell scripts, or Powershell and WinRM. You likely could, but it’d be much harder, would take several times more time, not to mention that a much larger portion of it would be untested. Ansible has become mature and reliable, and you can more or less rely on it doing what it promises. Its idempotence focus really does help, too. You can give points to people with Ansible on their resumé when hiring. This is not the case for your brand of hand-rolled shell scripts. The main value is that you get a consistent interface to all the regular sysadmin stuff on both Windows and Linux, networking equipment, and all sorts of REST and other APIs. You don’t have to check-then-do, you don’t need to mix sh</code> and Powershell, you don’t need to mess about with manual curl</code> calls, it’s done for you and it’s a huge time saver. You interact with it in the same way and handle it all at once. You might think that it’d be nice if there were a similar large collection of shell code, like libraries, that does the same things and you just call it from shell, but there isn’t. Yet Ansible exists, is widely used and has an incomparably large online presence, wealth of resources and thirdparty modules. I dare say it’s a strictly superior solution to just shell scripts, if you’re stuck with pet servers and infrastructure. Vs Infrastructure As Code counterparts</h2> There’s seemingly stiff competition from Puppet</a>, Saltstack</a>, Chef</a>, but they differ enough that it’s not really the case. Ansible is the fully-featured agentless solution. Salt has salt-ssh</code>, Puppet has Bolt</a>, neither are as fully featured as their agent-based counterparts. Not needing an agent installed on machines simplifies operation and makes drive-by configuration simple. Say, you want to ensure something is installed and configured on a set of machines, it’s quite easy to do. Being agentless, Ansible does not ensure it stays that way, of course. This is what defines its whole approach to configuration management. It is not as pure of an IaC tool as its competitors. You don’t define the complete machine configuration in code and have agents enforce it, you have some code ensuring something is in a certain state when you run it, not more and not less. This is a less powerful, but also requires way less commitment, which is what makes it so appealing. It takes very little effort in comparison to, say, Puppet, to get some initial results. Speed is not Ansible’s strength. It’s uses mostly SSH for transport, and even with TCP parameter tweaks, pipelining or Mitogen optimizations</a> it’s never fast for more complex jobs in high-ping scenarios (servers far away). You will be tempted to run it from a server closer to targets at some point. Ansible has by far the most active community, which gives you the best odds of finding an answer or a full solution for something online. You can find plugins for pretty much anything. This is something that is not to be discounted and is very important. LLM also speak it with proficiency. When it does / doesn’t make sense</h2> It makes sense when you need to ensure something is installed and configured on some pet machines. It’s a superior solution to doing everything with shell and SSH. Although I’ve seen it used like this, it doesn’t really make a difference just for provisioning a new VM (first thing you run after is). If you’re planning to run something only once, and in a single place, then it’s not much better than shell scripts. “Modern” approach: use a tool like Packer</a> or Goldboot</a> to create VM images, then deploy them as VMs with OpenTofu</a> or even to bare metal machines with PXE </aside> It doesn’t make sense when you want to really configure all your infra in code in a repo. In that case, Puppet is a better solution. However, I maintain that this is a legacy approach and the OpenTofu + Packer approach always makes more sense, even when you’re installing on bare metal. It may not cover all your needs (especially for Windows), and in that case you may use a bit of Puppet or Salt. A replacement?</h2> From what I’ve seen, there is none. I can hypothesize what improvments can be made in a potential replacement: it’s mostly addressing the points from above: non-global vars, real arguments, namespacing - all obvious improvements</li> simplification of how roles are defined and documented, move more of the semantics into the files and out of the paths, less duplication</li> better code reuse paradigm</li> overall simplification of role / task / playbook distinctions</li> </ul> Of course, any replacement does away with one of Ansible’s major strengths, which is widespread use and a huge amount of resources online. You can’t write that colletive experience into existence. Another consideration: Ansible is one of those tools that would never have taken off if it weren’t FOSS. Any replacement for it would have to match that. Examples</h2> Installing unnecessary utilities, zsh and Vim plugins from random repos and such, increases supply chain attack surface of your machines. Don’t do it on your servers </aside> For this blog post, I’ve ironed out a couple of playbooks</a> that install my dotfiles and some essential utilities onto Linux and Windows machines. For Windows machines the playbook also tweaks the GUI a bit, like adjusting taskbar settings, reverting questionable Windows 11 decisions, showing hidden files and such. I’m too lazy to rewrite it for this blog post, but here’s what a real life example of Ansible use would look like - configuring a Windows Gitlab runner for a legacy Windows project: installing the required tools for compilation and testing</li> [unregistering and] registering a new Gitlab runner with community.general.gitlab_runner</a> (registration is required to get a runner token)</li> installing the Gitlab runner into the machine, configuring it (includes templating the token into the config)</li> ensuring the Windows service for it exists, is started and set for autorestart</li> </ul> Other simple examples include installing Node and Windows Prometheus exporters, installing and configuring Postgres and Mongo exporters (including creating DB users for them) and such. Also patching pet database servers or legacy servers if you have those. A nice example someone else wrote would be Kubernetes node patching</a> though, again, I’d say that this sort of pet-sitting is inferior to just replacing the node with Terraform. Conclusion</h2> I’d say that Ansible is, in spite of its numerous shortcomings, a tool worth knowing and currently irreplaceable for its purpose: doing simple common tasks on multiple machines and interacting with APIs along the way. Its lack of purity shouldn’t discourage you from using it, because it’s better than the alternative (shell scripts). There are few roles where its competitors beat it that are not served better by a more modern approach. I guess it’s important to note that it’s a helper tool, and unlikely to be the focus of your business, so a laxer criteria applies compared to something that’s your main business focus. I do not suggest you use it in a more important role (deploying and managing the whole system) in 2025 and onwards, the modern OpenTofu (Terraform) + Packer [+ Kubernetes] combo has become the standard for a reason and is worth considering - it’s only hard if you’re not familiar with it, otherwise it’s all quite logical. Not having a human in the loop for redeploying unhealthy infra is a huge QoL improvement.



Tek scope screen capture with Bash
Unknown — Sat, 22 Jul 2023 00:00:00 +0000
I have a Tektronix TDS2014</span></a> oscilloscope
(4-channel, 100 MHz, 1
GS/s) from the early 2000s.
It’s useful for troubleshooting devices and while prototyping circuits.
I would also like to use screen captures from it in my blog posts.
I could always photograph the screen, but that goes against my
nature, and the results would be objectively bad because the screen is of the
era.</p>

I find protocol decoding on cheap oscilloscopes far inferior to using Sigrok
PulseView</a> on your computer with a USB</span> logic analyzer; you can even write your own decoders in
Python to decode device-specific protocols that lie on top of standard
protocols like SPI</span> or CAN</span></p>
</aside>
This scope is not favored by the community (small memory of only 2500 points,
no protocol decoding), but is portable, fast to boot up and has an intuitive
user
interface.
More importantly for this post, the TDS2MM</span></a> module which I got with the scope
provides an RS232</span> port (alongside GPIB</span>
and a Parallel port) I can use to connect to it.
Tektronix has some
software</a>
that allows you to interface with the scope, but it’s Windows-only and overkill
for my purposes.</p>
The scope allows for “printing”</em> in various formats to each of its outputs.
Apart from output compatible with various 90s Hewlett-Packard and Epson
printers, it offers BMP</span>, PCX</span>, TIFF</span>, RLE</span> and EPS</span> output.
Once you pick the output and format, the Print</em> button on the front panel
conveniently sends the data through the chosen port.</p>
Connecting the oscilloscope</h2>
The easiest and cheapest (but not highest-performance) way to interface with
your modern Linux computer is definitely a USB</span>-to-RS232</span> interface with a DE-9</span> port (watch out, you
may need a gender changer for it).
Figuring out RS232</span> communication schemes is usually easy,
and such interfaces are easy to work with in Linux.</p>
First, let’s out which RS232</span>
parameters (Baud rate, parity, etc.) the scope uses.
Pressing the Utility</em> button on the scope, then Options</em> > RS232 Setup</em>
brings us to the configuration screen which allows us to change some of these
parameters.
Obviously, I’d like the transfers to be as fast as possible, so I picked the
highest Baud rate available, and set the other parameters to personal taste
(and what seemed the easiest to work with).
These are the settings I ended up with:</p>
Baud</th> Flow control</th> EOL string</th> Parity</th></tr></thead>

19200</td> None</td> CR</span></td> Odd</td></tr>
</tbody></table>
Then we need to pick the output format. Pressing the Utility</em> button, then
Options</em> > Printer Setup</em> brings us to the configuration screen where we pick
EPSIMAGE</code> as the format, and obviously RS232</span> as the port.
EPS</span> (Encapsulated PostScript) promises a beautiful
vector-based output.</p>

I prefer picocom to minicom, it’s a more minimal and easier to use alternative</p>
</aside>
Let’s use picocom</code> to look at the output.
Since PostScript is a textual format (actually a Turing-complete language), we
need to map the EOL</span> string properly to see new lines.
Take note that picocom</code> actually requires both CR</span>
(return cursor to first column) and LF</span> (move cursor
down one line) to properly display the text (even though Linux usually uses
just LF</span>).
Setting the scope to output CR/LF</span> doesn’t seem to work, so
we must set it up to output CR</span>, then map that CR</span> into a CRLF</span> for display purposes
with --imap crcrlf</code>.</p>
picocom</span></span> \
</span></span></span>    --</span>baud</span> 19200 \
</span></span></span>    --</span>parity</span> o \
</span></span></span>    --</span>databits</span> 8 \
</span></span></span>    --</span>stopbits</span> 1 \
</span></span></span>    --</span>imap</span> crcrlf \
</span></span></span>    /dev/ttyUSB0</span>
</span></code></pre>
When we run picocom</code> and press Print</em> on the device, we get the PostScript code!</p>
EPS - saving into a file</h2>
Okay, now that we know that transfers work, we need to figure out how to save
the PostScript into a file.</p>
Looking at the output in picocom</code>, we can see that it begins with
%!PS-Adobe-2.0 EPSF-2.0</code> and ends with %%EOF</code>.
That should make it easy to know when the file was read to the end.
It’d also be nice if we could have the script listen for transfers, and save
each screen capture into a new file.</p>
Interacting with a tty</code> on Linux is a simple combination of setting the
appropriate tty</code> settings (can be done with stty</code> from shell) and
reading/writing the appropriate /dev/tty*</code> device file.
Doing it all from a shell requires no dependencies, so there’s no
reason to use anything but shell; Bash is a bit easier than POSIX</span> shell, so let’s go with that.</p>
#</span></span>!/usr/bin/env bash</span>
</span></span>
</span>set</span></span> -</span>o</span> errexit</span>
</span>set</span></span> -</span>o</span> pipefail</span>
</span>set</span></span> -</span>o</span> noclobber</span>
</span>set</span></span> -</span>o</span> nounset</span>
</span>
</span>DEVFILE</span>=</span>$</span>1</span></span></span>
</span>
</span>#</span></span> check if it's a character device</span>
</span></span>if</span> !</span> [</span> -</span>c</span> "</span>$</span>DEVFILE</span></span>"</span></span> ]</span></span>;</span> then</span>
</span>	echo</span></span> >&</span>2</span> not a character device</span>
</span>	exit</span></span> 1</span>
</span>fi</span>
</span>
</span>#</span></span> save stty state and restore it on exit</span>
</span></span>OLD_STTY</span>=</span>"</span>$</span>(</span>stty</span></span> --</span>save</span> --</span>file</span> "</span>$</span>DEVFILE</span></span>"</span></span></span>)</span></span>"</span></span></span>
</span>cleanup</span>(</span>)</span> {</span>
</span></span>	stty</span></span> --</span>file</span> "</span>$</span>DEVFILE</span></span>"</span></span> "</span>$</span>OLD_STTY</span></span>"</span></span></span> ||</span>
</span></span>		stty</span></span> --</span>file</span> "</span>$</span>DEVFILE</span></span>"</span></span> "</span>$</span>OLD_STTY</span></span>"</span></span></span>
</span></span>}</span></span>
</span>trap</span></span> cleanup SIGINT SIGTERM ERR EXIT</span>
</span>
</span>STTY_SETTINGS</span>=</span>"</span>19200 raw parenb parodd hupcl \
</span></span></span></span>    clocal icrnl -iexten -echo"</span></span></span>
</span>
</span>#</span></span> set stty state, repeat if it fails</span>
</span></span>#</span></span> shellcheck disable=SC2086</span>
</span></span>stty</span></span> --</span>file</span> "</span>$</span>DEVFILE</span></span>"</span></span> $</span>STTY_SETTINGS</span></span></span> ||</span>
</span>	stty</span></span> --</span>file</span> "</span>$</span>DEVFILE</span></span>"</span></span> $</span>STTY_SETTINGS</span></span></span>
</span>
</span>#</span></span> Set the device to:</span>
</span></span>#</span></span> 19200 baud</span>
</span></span>#</span></span> flow control: None</span>
</span></span>#</span></span> EOL string: CR</span>
</span></span>#</span></span> parity: odd</span>
</span></span>
</span>echo</span></span> "</span>Listening for Tektronix scope EPS"</span></span> \
</span></span></span>	"</span>output on '$</span>DEVFILE</span></span>'"</span></span></span>
</span>
</span>#</span></span> shellcheck disable=SC2002</span>
</span></span>cat</span></span> "</span>$</span>DEVFILE</span></span>"</span></span></span> |</span> while</span> true</span></span>;</span> do</span>
</span>	read</span></span> -</span>r</span></span>
</span>	#</span></span> Ignore data before the prologue</span>
</span></span>	[[</span> $</span>REPLY</span></span> !=</span> %</span></span>!</span>PS-*</span> ]]</span></span> &&</span> continue</span>
</span>
</span>	out</span>=</span>"</span>out_$</span>(</span>date</span></span> +%</span>F</span></span>.%</span>H</span></span>.%</span>M</span></span>.%</span>S</span></span></span>)</span></span>.ps"</span></span></span>
</span>	START_TIME</span>=</span>"</span>$</span>(</span>date</span></span> +%</span>s</span></span></span>)</span></span>"</span></span></span>
</span>
</span>	echo</span></span> "</span>capturing postscript to '$</span>out</span></span>'..."</span></span> 1</span>>&</span>2</span></span>
</span>	printf</span></span> "</span>."</span></span></span>
</span>	echo</span></span> "</span>$</span>REPLY</span></span>"</span></span> ></span>"</span>$</span>out</span></span>"</span></span></span>
</span>
</span>	while</span> true</span></span>;</span> do</span>
</span>
</span>		read</span></span> -</span>r</span> -t30</span> ||</span> {</span>
</span>			echo</span></span> "</span>read failed."</span></span> 1</span>>&</span>2</span></span>
</span>			break</span>
</span>		}</span>
</span>
</span>		printf</span></span> "</span>."</span></span></span>
</span>		echo</span></span> "</span>$</span>REPLY</span></span>"</span></span> >></span>"</span>$</span>out</span></span>"</span></span></span>
</span>		#</span></span> break if epilogue found</span>
</span></span>		[[</span> $</span>REPLY</span></span> ==</span> %</span>%</span></span>EOF ]]</span></span> &&</span> break</span>
</span>	done</span>
</span>	ELAPSED</span>=</span>$</span>((</span>$</span>(</span>date</span></span> +%</span>s</span></span></span>)</span></span> -</span> START_TIME))</span></span></span>
</span>	echo</span></span>
</span>	echo</span></span> "</span>capture done for file '$</span>out</span></span>',"</span></span> \
</span></span></span>		"</span>took $</span>ELAPSED</span></span> seconds"</span></span> 1</span>>&</span>2</span></span>
</span>done</span>
</span>
</span></code></pre>
The script detects the PostScript prologue and starts writing to a new file,
until it detects the epilogue.
The script also cleans up after itself, restoring the stty</code> values.</p>
After running the script and pressing the Print</em> button, it takes some 23
seconds to receive the screenshot. Not too bad, but not great. The resulting
file weighs around 25KB. We can now finally inspect the received EPS</span>. Opening it with
zathura</code> (zathura-ps</code> required) yields - disappointment.
Not a vector in sight. The file contains a bitmap of the screen, and it is in
the middle of the page.</p>

PostScript is the epitome of a write-only language, and Ghostscript
is one of the most ergonomically challenged commandline utilities I’ve ever
encountered</p>
</aside>
Inspecting the file, we find the actual bitmap part encoded as a hex string,
and a bit of PS</span> code that sets the page size and
displays the image.
That hex string seems wasteful (1 byte encodes to 2 hex bytes). Even base64
seems efficient in comparison (3 bytes encode to 4 base64 bytes), but I guess
that PS</span> only supports hex for saving bitmaps.</p>
It’s obvious that this is not the most space-efficient way of saving the
bitmap, and actually converting the PS</span> file into a
pixel-perfect bitmap in some more widespread format isn’t trivial, probably
probably requiring use of ghostscript</code>.
The size issue reflects on transfer speed.</p>
So, after all, it makes more sense to try receiving one of the binary bitmap
formats instead.</p>
Figuring out the best raster image format</h2>
Let’s take into consideration that the bitmap sent isn’t color-diverse and
is either 256-colors or some sort of indexed / palette color
</a>.
You can figure that out from the screen itself.
The screen resolution is 320×240, for reference.</p>
Let’s try to find what’s the optimal format among those offered by the
oscilloscope:</p>

RLE</span> is likely a run-length
encoded</a> bitmap; this is a
primitive, better-than-nothing compression algorithm</li>
PCX</span> has indexed colors and is RLE</span>-encoded</li>
TIFF</span> can have palette colors, and can support LZW</span></a>
compression, which is usually better than RLE</span></li>
BMP</span> is unlikely to be compressed, especially with
RLE</span> showing as a separate option</li>
</ul>

TIFF</span> is jokingly referred to as “Thousands of
Incompatible File Formats”</p>
</aside>
We don’t seem to be able to infer which format is the most space-efficient,
because we don’t know which exact variants of these formats are supported by
the oscilloscope.
What we can conclude is that all of these are binary formats.</p>
Let’s try just receiving captures in all of the formats, and then compare their
file sizes.
To capture any of the formats, we use the following trivial process:</p>

Use stty</code> to set the appropriate tty</code> parameters</li>
cat</code> the tty</code> device into a file</li>
Pick the format in the oscilloscope’s menu, press the Print</em> button to
start the transfer</li>
In parallel, watch that file with tail -f</code>, to see when the process stops</li>
Ctrl-C</em> the cat</code></li>
</ol>
Here are the commands:</p>
stty</span></span> -</span>F</span> /dev/ttyUSB0 19200 raw parenb parodd</span>
</span>cat</span></span> /dev/ttyUSB0 ></span> /tmp/tekrecv</span>
</span>#</span></span> in another terminal:</span>
</span></span>tail</span></span> -</span>f</span> /tmp/tekrecv</span>
</span></code></pre>
Running these commands, we get these files with wildly varying sizes (and
transfer times):</p>
Format</th> File size (KB)</th> Transfer time (s)</th></tr></thead>

BMP</span></a></td> 80</td> 44</td></tr>
PCX</span></a></td> 13</td> 8</td></tr>
TIFF</span></a></td> 17</td> 10</td></tr>
RLE</span></a></td> 16</td> 11</td></tr>
</tbody></table>
Seems like the PCX</span> file is the winner, and it’s 8
second transfer time is quick enough not to be intrusive.</p>
These formats are suboptimal for modern web publishing (and they are quite
large for what they are), but converting them to PNG is trivial with
ImageMagick:</p>
convert</span></span> capture.pcx capture.png</span>
</span></code></pre>

OptiPNG</a> is a PNG</span> size optimizer</p>
</aside>
The resulting file is a slender 2.6KB</span>. Running
optipng</code> on it only saves us another 100B</span>, so no
point in doing it. I’ve teased enough, here’s what the end result looks like:</p>

  
</div>

This blog contains JavaScript that proportionally resizes “pixel art” images to
the largest integer multiple of their width that fits the space</p>
</aside>
Quite charming, especially when combined with image-rendering: pixelated</code> CSS property and with size being an integer multiple of the original
size.</p>
Note the Ink Saver: On</em> option in the screen grab: it’s responsible for this
quite elegant color scheme.
Otherwise, the default color scheme has a saturated blue + black background
that would have surely inspired me to palette-swap it.
I won’t desecrate this blog and post a screenshot.</p>
Capturing raster images with a script</h2>
It’d be nice to have a script which automatically receives the image and saves
it into a file, without having to manually look at the tail -f</code> output.</p>
My first intuition was to try and figure out how many bytes each transfer would
contain, and then dd</code> it into a file.
These binary file formats usually have a fixed-length header which contains the
size of the image, which you use to figure out how many bytes you need to read.
We could use dd</code> to read that size, and then use dd</code> again to read the rest
of the file.</p>
Then it dawned on me: the transfer from the oscilloscope is a continuous,
uninterrupted stream of bytes. When the stream stops for a while, we can safely
infer that the transfer is complete, just like we did while initially receiving
the files.</p>
I tried searching for a tool online that allows us to stop a transfer (or
pipeline) when no bytes appear on the input for a while, to no avail.
PySerial seems to have such an option, but that’s kind of overkill.</p>
When we do this manually, we cat</code> the tty</code> and look at the tail -f</code> output
of the destination file to see when it stops changing.
How can we do something similar programatically?
By looking at the mtime</code> of the destination file in a loop.
When it stops changing for a while (the timeout), the file transfer is done and
we can kill</code> the cat</code> by PID</span>.
We let the cat</code> run in the background beforehand.</p>
I’ve added a couple of conveniences to the script that implement this idea:</p>

It runs continuously and creates a new file for each new transfer</li>
It shows the elapsed time during transfers</li>
There’s a cleanup block that restores stty</code> settings, kills any leftover
processes, removes temp files</li>
The file names are automatically generated with timestamps</li>
We infer the extension from the file’s magic number using file</code></li>
If ImageMagick is available, we automatically convert the output to PNG</span></li>
</ul>
Finally, here’s the script. I hope it’ll inspire you to use Bash for similar
jobs, it fits this role competently.
You can take Bash too far and create an unmaintainable mess (and some people
love doing that), but tasks like this are right in the middle of its comfort
zone.</p>
#</span></span>!/bin/bash</span>
</span></span>
</span>SERIAL_PORT</span>=</span>"</span>$</span>{</span></span>1</span></span>:-</span></span>"</span>/dev/ttyUSB0"</span></span></span>}</span></span>"</span></span></span>
</span>TIMEOUT</span>=</span>$</span>{</span></span>2</span></span>:-</span></span>1</span>}</span></span></span>
</span>
</span>#</span></span> check if it's a character device</span>
</span></span>if</span> !</span> [</span> -</span>c</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> ]</span></span>;</span> then</span>
</span>	echo</span></span> >&</span>2</span> not a character device</span>
</span>	exit</span></span> 1</span>
</span>fi</span>
</span>
</span>#</span></span> Save the current stty settings to restore on exit</span>
</span></span>OLD_STTY</span>=</span>"</span>$</span>(</span>stty</span></span> --</span>save</span> --</span>file</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span></span>)</span></span>"</span></span></span>
</span>#</span></span> Configure the serial port</span>
</span></span>STTY_SETTINGS</span>=</span>"</span>19200 raw parenb parodd hupcl \
</span></span></span></span>    clocal -iexten -echo"</span></span></span>
</span>#</span></span> shellcheck disable=SC2086</span>
</span></span>stty</span></span> --</span>file</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> $</span>STTY_SETTINGS</span></span> \
</span></span></span></span>    ||</span> stty</span></span> --</span>file</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> $</span>STTY_SETTINGS</span></span></span>
</span>
</span>#</span></span> Continually read from the serial port,</span>
</span></span>#</span></span> create new file for each transfer</span>
</span></span>while</span> true</span></span>;</span> do</span>
</span>    #</span></span> Create a temporary file</span>
</span></span>    TEMPFILE</span>=</span>"</span>$</span>(</span>mktemp</span></span>)</span></span>"</span></span></span>
</span>
</span>    #</span></span> Start reading from the serial port into the</span>
</span></span>    #</span></span> temporary file</span>
</span></span>    cat</span></span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> ></span> "</span>$</span>TEMPFILE</span></span>"</span></span></span> &</span>
</span>    CAT_PID</span>=</span>$</span>!</span></span></span>
</span>
</span>    #</span></span> Define a cleanup function</span>
</span></span>    cleanup</span>(</span>)</span>
</span></span>              {</span>
</span></span>        kill</span></span> "</span>$</span>CAT_PID</span></span>"</span></span> 2</span>></span> /dev/null</span>
</span></span>        rm</span></span> -</span>f</span> "</span>$</span>TEMPFILE</span></span>"</span></span></span>
</span></span>        #</span></span> Restore stty settings</span>
</span></span></span>        stty</span></span> --</span>file</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> "</span>$</span>OLD_STTY</span></span>"</span></span> \
</span></span></span></span></span>            ||</span> stty</span></span> --</span>file</span> "</span>$</span>SERIAL_PORT</span></span>"</span></span> "</span>$</span>OLD_STTY</span></span>"</span></span></span>
</span></span>    }</span></span>
</span>    trap</span></span> cleanup SIGINT SIGTERM ERR EXIT</span>
</span>
</span>    echo</span></span> "</span>Waiting for file transfer..."</span></span> >&</span>2</span></span>
</span>    #</span></span> Wait until there's data in the temporary file</span>
</span></span>    while</span> [[</span> !</span> -</span>s</span> $</span>TEMPFILE</span></span> ]]</span></span>;</span> do</span>
</span>        sleep</span></span> 0.1</span>
</span>    done</span>
</span>
</span>    echo</span></span> "</span>Starting file transfer..."</span></span> >&</span>2</span></span>
</span>    START_TIME</span>=</span>"</span>$</span>(</span>date</span></span> +%</span>s</span></span></span>)</span></span>"</span></span></span>
</span>
</span>    #</span></span> Monitor the temporary file's size, stop when</span>
</span></span>    #</span></span> it stops being modified</span>
</span></span>    while</span> true</span></span>;</span> do</span>
</span>        #</span></span> calculate time for showing progress</span>
</span></span>        CURRENT_TIME</span>=</span>$</span>(</span>date</span></span> +%</span>s</span></span></span>)</span></span></span>
</span>        ELAPSED_TIME</span>=</span>$</span>((</span>CURRENT_TIME -</span> START_TIME))</span></span></span>
</span>
</span>        SECONDS</span>=</span>$</span>((</span>ELAPSED_TIME %</span> 60</span>))</span></span></span>
</span>        MINUTES</span>=</span>$</span>((</span>ELAPSED_TIME /</span> 60</span> %</span> 60</span>))</span></span></span>
</span>        HOURS</span>=</span>$</span>((</span>ELAPSED_TIME /</span> 3600</span>))</span></span></span>
</span>
</span>        ELAPSED_TIME_STR</span>=</span>"</span>$</span>(</span>printf</span></span> \
</span></span></span></span></span></span>            "</span>%02d:%02d:%02d"</span></span> \
</span></span></span></span></span></span>            $</span>HOURS</span></span> $</span>MINUTES</span></span> $</span>SECONDS</span></span></span>)</span></span>"</span></span></span>
</span>
</span>        #</span></span> take note of output file</span>
</span></span>        #</span></span> mtimes before and after pause</span>
</span></span>        LMTIME</span>=</span>$</span>(</span>stat</span></span> -</span>c</span> %</span>Y</span></span> "</span>$</span>TEMPFILE</span></span>"</span></span></span>)</span></span></span>
</span>        sleep</span></span> "</span>$</span>TIMEOUT</span></span>"</span></span></span>
</span>        NMTIME</span>=</span>$</span>(</span>stat</span></span> -</span>c</span> %</span>Y</span></span> "</span>$</span>TEMPFILE</span></span>"</span></span></span>)</span></span></span>
</span>
</span>        #</span></span> If the file hasn't been modified for</span>
</span></span>        #</span></span> $TIMEOUT long, kill the cat,</span>
</span></span>        #</span></span> move / convert the file</span>
</span></span>        if</span> [[</span> $</span>LMTIME</span></span> ==</span> "</span>$</span>NMTIME</span></span>"</span></span> ]]</span></span>;</span> then</span>
</span>            kill</span></span> $</span>CAT_PID</span></span></span>
</span>
</span>            #</span></span> Form the output file name using the date</span>
</span></span>            OUTPUT_FILE</span>=</span>"</span>out_$</span>(</span>date</span></span> +%</span>F_</span></span>%</span>H</span></span>.%</span>M</span></span>.%</span>S</span></span></span>)</span></span>"</span></span></span>
</span>
</span>            FILE_TYPE</span>=</span>"</span>$</span>(</span>file</span></span> -</span>b</span> --</span>mime-type</span> "</span>$</span>TEMPFILE</span></span>"</span></span></span>)</span></span>"</span></span></span>
</span>            EXTENSION</span>=</span>"</span>"</span></span></span>
</span>
</span>            case</span> "</span>$</span>FILE_TYPE</span></span>"</span></span> in</span>
</span></span>                </span>image/bmp)</span></span></span> EXTENSION</span>=</span>"</span>bmp"</span></span></span> </span>;;</span></span>
</span></span>                </span>image/x-pcx)</span></span></span> EXTENSION</span>=</span>"</span>pcx"</span></span></span> </span>;;</span></span>
</span></span>                </span>image/tiff)</span></span></span> EXTENSION</span>=</span>"</span>tiff"</span></span></span> </span>;;</span></span>
</span></span>                </span>*</span>)</span></span></span> EXTENSION</span>=</span>"</span>bin"</span></span></span> </span>;;</span></span>
</span></span>            esac</span></span>
</span>
</span>            mv</span></span> "</span>$</span>TEMPFILE</span></span>"</span></span> "</span>$</span>OUTPUT_FILE</span></span>.$</span>EXTENSION</span></span>"</span></span></span>
</span>
</span>            if</span> command</span></span> -</span>v</span> convert ></span> /dev/null</span>;</span> then</span>
</span>                convert</span></span> "</span>$</span>OUTPUT_FILE</span></span>.$</span>EXTENSION</span></span>"</span></span> \
</span></span></span>                    "</span>$</span>OUTPUT_FILE</span></span>.png"</span></span></span>
</span>            fi</span>
</span>
</span>            printf</span></span> "</span>\r"</span></span></span>
</span>            echo</span></span> "</span>Finished file transfer:"</span></span> \
</span></span></span>                "</span>$</span>OUTPUT_FILE</span></span>.$</span>EXTENSION</span></span> in"</span></span> \
</span></span></span>                "</span>$</span>ELAPSED_TIME_STR</span></span>"</span></span> >&</span>2</span></span>
</span>            echo</span></span>
</span>            break</span>
</span>        else</span>
</span>            printf</span></span> "</span>\r%s"</span></span> "</span>$</span>ELAPSED_TIME_STR</span></span>"</span></span> >&</span>2</span></span>
</span>        fi</span>
</span>    done</span>
</span>done</span>
</span></code></pre>
I look forward to using some oscilloscope screenshots in future posts.
Finally, no such article would be complete without a cheesy Lissajous pattern.</p>

  
</div>