Jekyll2019-09-29T10:17:10+10:00https://blog.sigsec.net/feed.xmlsigsecA blog about tinkering with technology, pulling things apart, and finding out what I can put together. I like documenting my projects and thoughts here, both for my own benefit and to help others who have similar interests.sigsecTwo-factor TOTP codes2019-09-28T20:59:09+10:002019-09-28T20:59:09+10:00https://blog.sigsec.net/posts/2019/09/two-factor-totp-codes<p>Two-Factor authentication is a good way of increasing login security in systems, and is now quite widespread: it’s the six digit number that rolls over every 30 seconds. It’s designed to be very simple to operate as a user, but how much effort is involved as a developer?
<!--more--></p>
<h2 id="how-2fa-works">How 2FA works</h2>
<p>There’s a couple of different methods for multi-factor authentication; I’ll be looking at the <a href="https://en.wikipedia.org/wiki/Time-based_One-time_Password_algorithm">Time-based One-Time Password algorithm</a> (TOTP).</p>
<p>As it turns out, it’s quite straight-forward to get started with. On the server side, a secret key is generated and stored, then sent to the user (typically as a QR code or a hex string). This is a <a href="https://en.wikipedia.org/wiki/Shared_secret">shared secret</a>.</p>
<p>When the user wants to login, the server asks for their code. The user generates this from the shared secret we gave them earlier and the current time. Concurrently, the server generates a code in the same way. If both system clocks are in sync, the codes match and the user is validated.</p>
<p>It’s quite common for clocks to be slightly out (potentially up to a minute each way), so the server can compensate by generating additional codes that have been shifted in time; if one of these codes lines up with the user, we can still validate them.</p>
<p class="notice--info">This is just an overview — in practice, you’ll need to deal with more complexities (such as not allowing the same 2FA code to be submitted multiple times).</p>
<h2 id="developer-implementation">Developer implementation</h2>
<h3 id="in-python">In Python</h3>
<p>Python has the very handy <a href="https://github.com/pyauth/pyotp">PyOTP library</a>, which takes care of pretty much everything for us.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pyotp</span>
<span class="c1"># Generate a new shared secret (or load one made previously)
</span><span class="n">shared_secret</span> <span class="o">=</span> <span class="n">pyotp</span><span class="o">.</span><span class="n">random_base32</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="s">"The TOTP seed is: </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">shared_secret</span><span class="p">)</span>
<span class="c1"># Make a TOTP object and get the current verification code
</span><span class="n">totp</span> <span class="o">=</span> <span class="n">pyotp</span><span class="o">.</span><span class="n">TOTP</span><span class="p">(</span><span class="n">shared_secret</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"The TOTP code is: </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">totp</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
</code></pre></div></div>
<p>We can also generate a QR code to send to the user through the <a href="https://github.com/mnooner256/pyqrcode">pyqrcode library</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pyqrcode</span>
<span class="n">uri</span> <span class="o">=</span> <span class="n">totp</span><span class="o">.</span><span class="n">provisioning_url</span><span class="p">(</span><span class="s">"user@example.com"</span><span class="p">,</span> <span class="s">"Cool App"</span><span class="p">)</span>
<span class="n">qr</span> <span class="o">=</span> <span class="n">pyqrcode</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
<span class="n">qr</span><span class="o">.</span><span class="n">png</span><span class="p">(</span><span class="s">"output.png"</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="in-javascript">In JavaScript</h3>
<p class="notice--info">I’m using this in a Browser context, so you’ll need to make changes if you’re using it inside Node.</p>
<p>JavaScript is the same: nothing built-in, but the <a href="https://github.com/yeojz/otplib">otplib library</a> supports everything we need. I’m also using the <a href="https://github.com/soldair/node-qrcode">node-qrcode library</a> to make the QR code you’ll see below. Both of these libraries are loaded into the page using the <a href="https://unpkg.com/">unpkg CDN</a>. With that done, we can generate OTP codes:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Create a shared secret</span>
<span class="kd">var</span> <span class="nx">seed</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">otplib</span><span class="p">.</span><span class="nx">authenticator</span><span class="p">.</span><span class="nx">generateSecret</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">totp_seed</span><span class="dl">'</span><span class="p">).</span><span class="nx">value</span> <span class="o">=</span> <span class="nx">seed</span><span class="p">;</span>
<span class="c1">// Get the current token for the secret</span>
<span class="kd">var</span> <span class="nx">otp</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">otplib</span><span class="p">.</span><span class="nx">authenticator</span><span class="p">.</span><span class="nx">generate</span><span class="p">(</span><span class="nx">seed</span><span class="p">);</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">totp_token</span><span class="dl">'</span><span class="p">).</span><span class="nx">value</span> <span class="o">=</span> <span class="nx">otp</span><span class="p">;</span>
<span class="c1">// Show the QR code in the 'totp_qr' <canvas> tag.</span>
<span class="kd">var</span> <span class="nx">uri</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">otplib</span><span class="p">.</span><span class="nx">authenticator</span><span class="p">.</span><span class="nx">keyuri</span><span class="p">(</span>
<span class="dl">'</span><span class="s1">user@example.com</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">sigsec</span><span class="dl">'</span><span class="p">,</span> <span class="nx">seed</span><span class="p">);</span>
<span class="nx">QRCode</span><span class="p">.</span><span class="nx">toCanvas</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">totp_qr</span><span class="dl">'</span><span class="p">),</span> <span class="nx">uri</span><span class="p">);</span>
</code></pre></div></div>
<h2 id="my-examples">My examples</h2>
<p>I’ve written a couple of example implementations: if you’ve got any feedback or commentary, please <a href="https://twitter.com/sigsecdev">tweet at me</a> or <a href="/pages/about.html">drop me an email</a>!</p>
<h3 id="python-script">Python script</h3>
<p>I’ve written a handy little Python TOTP tool that you can grab from my <a href="https://github.com/sigsec/blog-examples/blob/master/totp-tool.py">Github here</a>. It should be a good starting point to see how to work with the two libraries I described above. It’s also fully functional, so feel free to put it through it’s paces!</p>
<figure class="half ">
<a href="/assets/images/totp-tool-help.png" title="The script includes detailed help information">
<img src="/assets/images/totp-tool-help.png" alt="A Linux terminal showing command help output" />
</a>
<a href="/assets/images/totp-tool-qr.png" title="The script in action, with a scannable QR code and the current token">
<img src="/assets/images/totp-tool-qr.png" alt="A Linux terminal showing command QR code output" />
</a>
<figcaption>My TOTP tool in action
</figcaption>
</figure>
<h3 id="live-javascript-demo">Live JavaScript demo</h3>
<p>Here’s a live demo, using JavaScript. This is only for fun and to play around with; please don’t try using this code in a production environment!</p>
<form>
TOTP seed:<input type="text" id="totp_seed" maxlength="16" /><br />
Code: <input type="text" id="totp_token" readonly="readonly" /><br />
Generated at: <input type="text" id="totp_time" readonly="readonly" /><br />
<input type="button" value="Generate new seed" onclick="newSeed()" /><br />
<div style="text-align: center">
<canvas id="totp_qr"></canvas>
</div>
</form>
<style type="text/css">
#totp_qr > img { display: inline !important; }
</style>
<script src="https://unpkg.com/otplib@^10.0.0/otplib-browser.js"></script>
<script src="https://unpkg.com/qrcode@1.4.2/build/qrcode.min.js"></script>
<script type="text/javascript">
function newSeed() {
var seed = window.otplib.authenticator.generateSecret(10);
document.getElementById('totp_seed').value = seed;
updateCode();
}
function updateCode() {
var seed = document.getElementById('totp_seed').value;
if(seed.length != 16) {
document.getElementById('totp_token').value = "Seed must be 16 chars";
return;
}
document.getElementById('totp_time').value = new Date().toLocaleTimeString();
var otp = window.otplib.authenticator.generate(seed);
document.getElementById('totp_token').value = otp;
var uri = window.otplib.authenticator.keyuri('user@example.com', 'sigsec', seed);
QRCode.toCanvas(document.getElementById('totp_qr'), uri);
}
var totpTimer = window.setInterval(updateCode, 1000);
newSeed();
</script>sigsecTwo-Factor authentication is a good way of increasing login security in systems, and is now quite widespread: it’s the six digit number that rolls over every 30 seconds. It’s designed to be very simple to operate as a user, but how much effort is involved as a developer?A holiday in Bali2019-09-24T21:47:41+10:002019-09-24T21:47:41+10:00https://blog.sigsec.net/posts/2019/09/a-holiday-in-bali<p>I recently flew to Bali for a holiday with my partner and some of my family. It was a really good trip for me to unwind, relax, and take some time away from work. Even though I wasn’t <em>working</em>, I was still thinking, and there were a few things that stood out to me during my travels.
<!--more--></p>
<p class="notice--info">In a lot of ways, this post both mirrors and is influenced by Derek Sivers’ blog post “<a href="https://sivers.org/how-was-india">How Was India?</a>”. Many of the things he comments on from his travels are things that I noticed in mine.</p>
<h2 id="what-i-noticed-in-bali">What I noticed in Bali</h2>
<h3 id="traffic">Traffic</h3>
<p>An easy one to start with — the traffic in Bali is very different to Australia. I’ve heard that most South-East Asian countries are similar, but my experience is limited to a few weeks in Vietnam a decade ago.</p>
<p>Getting out of the airport landed us right in the thick of it. I honestly loved it; seeing the <em>reasonably</em> orderly queue of cars grow and split into nine lanes for toll collection… then completely devolve and organically reassemble into one lane was bizarre but awesome. As we travelled further, more motorbikes were added to the mix until cars were outnumbered ten or more to one.</p>
<p>Road rules were merely a suggestion: one-way streets had bikes filtering against the flow of traffic; the only speed limit signs I saw were on the brand new motorway right next to the airport; of the traffic lights that existed, most flashed amber and were ignored. Overall, it felt very much like it was a self-organising system.</p>
<h3 id="enterprising-nature">Enterprising nature</h3>
<p>In general, it seems like the Balinese are an extremely enterprising group of people. Obviously a large portion of their income is derived from tourism-related activities, but the way this manifests itself is quite impressive: it seems like every tiny alleyway or place that <em>could</em> have a shop front, <em>does</em> have a shop front, and often there’s two or more.</p>
<p>There seems to be a few different categories of products they sell (surfboards with rude slogans and carved wooden penises were both popular) but there’s obviously a market for this or they wouldn’t be advertising them. Mimco-branded bags, phone cases and clutches also seem like a big seller, because they were in abundance everywhere I went.</p>
<p>I was surprised when, a couple of days into the trip, I noticed road-side stands with vodka bottles (typically Absolut) full of petrol. At first I was in some disbelief — the very first ones I saw didn’t have screw-tops, but instead had flowers; so surely this was just a decorative display or something. Nope. It was a petrol station.</p>
<h3 id="urban-jungle">Urban jungle</h3>
<p>The jungle reclaims everything pretty quickly. We passed a lot of developments that were overgrown with weeds or choked with vines; it doesn’t take much. This is why the big resorts and hotels are always doing maintenance, it’s critical given the antagonistic environment.</p>
<p>The city was dense, and I didn’t need to travel far to see dozens of homes and shops blending together. At the same time, there were areas with an abundance of space. I had a weird sensation of vertigo when I realised that just one of the pools in the resort I was in was more than twice as big as an entire block of land in the suburbs where I grew up.</p>
<h2 id="notes-for-future-travel">Notes for future travel</h2>
<h3 id="research-and-preparation">Research and preparation</h3>
<p>Sometimes it’s a lot of fun to turn up somewhere new, without any initial research or preconceived notions. For me, I’d say this is not a bad thing when I’m on an organised trip covering a lot of destinations. However, when I’m going to just the one country… I need to do my research.</p>
<p>For this trip, I knew the names of my hotels, but I didn’t think to look at them on a map and work out the rough distances and times it would take to travel between them and the airport. I also hadn’t done my due diligence in looking up popular tourist destinations, so I could be proactive about seeing points of interest.</p>
<p>I’m glad I’d used Google Maps to download sections of the map for offline use. Knowing that I could always find my way back to the hotel as long as I had a GPS signal was very reassuring whenever I set out in an unknown direction.</p>
<h3 id="local-currency">Local currency</h3>
<p>Last year I travelled to Singapore and Malaysia, and didn’t get any local currency until it was nearly too late (taxis in Malaysia are cash only!). I’m far too used to living in the nigh-cashless society that Australia is becoming, where it’s possible to just use contactless payments from your credit card or phone to pay for basically everything.</p>
<p>However, I’ve rapidly learned that Australia is actually quite ahead of the pack in terms of its adoption of contactless payments. When I was in Europe, many places accepted cards but not contactless, and in London the contactless limit was set at £30 (compared to $100 domestically).</p>
<p>As a result, I’ve learned it’s generally a good idea to get some local currency out whenever I travel. My limited experience also suggests you’ll get a better exchange rate doing it upon your arrival in the foreign country, rather than locally before you leave.</p>
<h3 id="above-all">Above all</h3>
<p>But above all, my notes for future travel are:</p>
<ul>
<li>Get out and see what there is to offer</li>
<li>Don’t be afraid — most of the risks I worry about are from me blowing things out of proportion</li>
<li>Have fun, and learn from the experience.</li>
</ul>sigsecI recently flew to Bali for a holiday with my partner and some of my family. It was a really good trip for me to unwind, relax, and take some time away from work. Even though I wasn’t working, I was still thinking, and there were a few things that stood out to me during my travels.Parametric 3D design2019-07-06T17:39:51+10:002019-07-06T17:39:51+10:00https://blog.sigsec.net/posts/2019/07/parametric-3d-design<p>I mentioned in my <a href="/posts/2019/06/my-first-time-with-a-3d-printer.html">last post</a> that we’d had a 3D printer at work for a couple of months. In that time there’s been a lot of hype and printing, with many custom designs floating around. Most of my colleagues have used TinkerCAD, but I’ve had a lot of fun (and some headaches) with OpenSCAD.
<!--more--></p>
<h2 id="what-is-openscad">What is OpenSCAD?</h2>
<p><a href="https://www.openscad.org/">OpenSCAD</a> is a scripting-based 3D modelling package. While it has a render view, unlike most other modelling packages you’re not able to drag around parts or edit them graphically: everything has to be done through the code. If you’ve ever had to typeset scientific documents, you may be familiar with <a href="https://www.latex-project.org/">LaTeX</a> — a good comparison (in my opinion) is that OpenSCAD is the LaTeX of the modelling world.</p>
<p>It’s definitely harder to get started with than other modelling software, but the scripting nature really appeals to me. I like the ability to put the code into Git for version control, and that I can completely adjust my models in seconds just by changing a few key parameters.</p>
<p>This means that I could in theory spend a bit more time making the model so it is customisable, but then just generate as many different output files in whatever configuration is desired for family and friends.</p>
<h2 id="what-have-i-modelled">What have I modelled?</h2>
<h3 id="e3d-v6-nozzle">E3D V6 nozzle</h3>
<p>The <a href="https://e3d-online.com/brass-nozzles-v6">E3D V6 nozzle</a> is the tip of the hot-end used in Prusa i3 MK3 3D printers. It’s a pretty iconic shape, and the <a href="https://e3d-online.dozuki.com/c/V6_nozzles">mechanical drawing</a> is freely available online. This made it a pretty good practice piece for me to use while learning OpenSCAD.</p>
<figure class="centre">
<img alt="OpenSCAD interface, showing a render of the E3D V6 nozzle and part of the code that generates it" src="/assets/images/openscad-e3d-v6.png" />
</figure>
<p>This model is parametric — there’s a few key variables at the top of the file that can be redefined to adjust the inlet (filament) and tip geometries. The default parameters are for a 1.75 mm filament and a 0.4 mm tip, the same as the nozzle that comes with the Prusa kit.</p>
<h3 id="radiation-symbol-cookie-cutter">Radiation symbol cookie cutter</h3>
<p>My partner is a radiation therapist, so I thought it would be nice to model and print a radiation symbol cookie cutter that I could then use for her birthday.</p>
<figure class="centre">
<img alt="OpenSCAD interface, showing a render of the radiation symbol cookie cutter and part of the code that generates it" src="/assets/images/openscad-cookiecutter-radiation.png" />
</figure>
<p>I printed two sizes, a 50 mm diameter version and an 80 mm diameter. When I went to use them for the first time, I found there were a couple of issues. Firstly, pressing cookie dough down to flatten it tended to make it very dense and hard when cooked, rather than light and fluffy.</p>
<figure class="centre">
<img alt="Radiation symbol cookies, raw and ready to bake" src="/assets/images/openscad-cookiecutter-radiation-raw.jpg" />
</figure>
<p>Secondly, the small gap between the edge of the cookie and the indent on the radiation wedges tended to make the wall fall away, rather than remaining upright. The last problem was, of course, that I simply overcooked them.</p>
<figure class="centre">
<img alt="Radiation symbol cookies, baked a bit too long" src="/assets/images/openscad-cookiecutter-radiation-cooked.jpg" />
</figure>
<h3 id="keys">Keys</h3>
<p>I’ve also modelled and printed… keys. With an interest in physical security, a box of old locks, and curiosity, I wanted to see how hard it would be to model and print keys from scratch. As it turns out, it’s <em>worryingly easy</em>.</p>
<p>Some $20 vernier calipers from the local hardware supply shop and a couple of hours of messing around in OpenSCAD produced the magnificent result below. Which is great and all, because it’s just a pretty picture, right? They’re purely decorative and won’t be able to harm anyone.</p>
<figure class="centre">
<img alt="OpenSCAD interface, showing a render of a key and part of the code that generates it" src="/assets/images/openscad-key.png" />
</figure>
<p>Not quite. As I had made it parametric, I could punch in the bitting code for any key that matched this profile. And they work. Ten minutes in the printer, a minute or so for the print to cool down, and then the keys just work in the lock without any post-processing steps.</p>
<p>Not that any of this is new. There <a href="https://www.thingiverse.com/thing:8925">are lots</a> of <a href="https://www.thingiverse.com/thing:162932">examples</a> of <a href="https://www.thingiverse.com/thing:9918">parametric key designs</a> on Thingiverse <a href="https://www.thingiverse.com/thing:56795">already</a>. It was just a surprise to me how easy it was to create my own.</p>
<h3 id="plug-followers">Plug followers</h3>
<p>Continuing on the theme of locksport (aka “lock picking”), I modelled and printed some plug followers. These are used when you’re disassembling and ‘de-coring’ a lock, to keep the upper pins and springs in place while you change the key pins. Essentially they’re just a cylinder, but I got some practice with conditionals in OpenSCAD by making the top and bottom profiles parametric.</p>
<figure class="centre">
<img alt="OpenSCAD interface, showing a render of a plug follower and the code that generates it" src="/assets/images/openscad-plug-follower.png" />
</figure>
<h3 id="other-stuff">Other stuff</h3>
<p>I made an ‘NFC token’ using one of the round NFC stickers I bought for a <a href="/posts/2019/04/reading-and-writing-nfc-tags.html">previous blog post</a>. It was pretty straight-forward to model, just a short flat cylinder with a smaller cylinder subtracted from the middle of it. I didn’t even do anything smart with the slicing to make it pause; I just watched it and hit pause in OctoPrint when it started to make the hole.</p>
<p>I made a name-tag for my partner at her work; this was a fun one, because I wanted to include her employer’s logo. I couldn’t find any vector graphic versions of it online, so I resorted to downloading a full-colour JPG of it and processing it in GIMP and then Inkscape to get the outlines as a DXF that OpenSCAD could then import.</p>
<p>There’s been a couple of other things I’ve designed here and there too, but not really much worth writing about.</p>
<h2 id="whats-next">What’s next</h2>
<p>Prusa3D have a sale on at the moment to celebrate hitting 100,000 sold printers, offering free international shipping on all Original Prusa i3 MK3S kits. This was enough incentive for me to stop <em>thinking</em> about buying one and to instead <em>actually buy it</em>, so my next post will likely be looking at what I ordered (and maybe how assembling it went, depending on how soon it arrives).</p>sigsecI mentioned in my last post that we’d had a 3D printer at work for a couple of months. In that time there’s been a lot of hype and printing, with many custom designs floating around. Most of my colleagues have used TinkerCAD, but I’ve had a lot of fun (and some headaches) with OpenSCAD.My first time with a 3D printer2019-06-29T21:58:09+10:002019-07-04T11:38:08+10:00https://blog.sigsec.net/posts/2019/06/my-first-time-with-a-3d-printer<p>Recently, work bought a 3D printer for prototyping and to encourage staff innovation and creativity. It’s been quite fun playing around with it and seeing designs I’ve found and made come to life. This post is going to be a quick recap of what I’ve printed lately.
<!--more--></p>
<h2 id="ordering-the-printer">Ordering the printer</h2>
<h3 id="printer">Printer</h3>
<p>The printer that work ordered is an <a href="https://shop.prusa3d.com/en/3d-printers/180-original-prusa-i3-mk3-kit.html">Original Prusa i3 MK3S</a>.</p>
<p>Many years ago I was planning to build a <a href="https://reprap.org/wiki/Prusa_Mendel">RepRap Prusa Mendel</a>, so it was quite interesting to see how the design of 3D printers has changed over time. The triangular frame of 8mm all-thread rod is gone, replaced by the more sturdy and extensible aluminium extrusions and frame pieces.</p>
<p>The control panel with an LCD and an SD-card slot also makes it a breeze to get started with. We’ve now set up OctoPrint though, so we can monitor jobs remotely and make it easier for other users to get their jobs ready.</p>
<h3 id="filaments">Filaments</h3>
<p>We also picked up a few different filaments to test with:</p>
<ul>
<li><strong><a href="https://shop.prusa3d.com/en/filament/159-pla-extrafill-gold-happens-750g.html">PLA in gold</a>:</strong> lovely prints with a lot of shimmer and sparkle in them, but was more expensive than the other PLAs.</li>
<li><strong>PLA in silver:</strong> we had some issues with under-extrusion on this one, but solid otherwise. I’m not sure which specific product this is for linking purposes, as it came included in the kit.</li>
<li><strong><a href="https://shop.prusa3d.com/en/filament/44-orange-pla-filament-1kg.html">PLA in neon orange</a>:</strong> really nice quality prints, but a bit bright for some uses.</li>
<li><strong><a href="https://shop.prusa3d.com/en/filament/111-pla-extrafill-noble-blue-750g.html">PLA in blue</a>:</strong> solid prints with no issues, but a bit dark for some uses.</li>
<li><strong><a href="https://shop.prusa3d.com/en/prusament/928-prusament-petg-neon-green-transparent-1kg.html">PETG in translucent green</a>:</strong> I wasn’t originally much of a fan of this one, but it’s grown on me. The material is more flexible and robust than PLA, and it’s cool to see the infill pattern in your object (but only if it’s a good one!).</li>
<li><strong><a href="https://shop.prusa3d.com/en/filament/130-white-easyabs-filament-1kg.html">ABS in white</a>:</strong> we only printed one small sample object with this and it didn’t really work for us; the material shrunk and warped shortly after printing. We’ll need to investigate it further.</li>
</ul>
<h2 id="what-ive-printed">What I’ve printed</h2>
<h3 id="coasters">Coasters</h3>
<p>On Thingiverse, user <em>jmdbcool</em> has uploaded the awesome <a href="https://www.thingiverse.com/thing:3027141">Infill coasters</a> design, along with comprehensive instructions on how to set up the patterns to your own specification.</p>
<p>This was a great way to test out the various filaments we’d ordered and to get a better idea of the look and physical characteristics of the infill patterns. They’re also quite quick to print, with an average print time of just over an hour (based on a 0.2mm layer height and infill densities of 10% or 20%).</p>
<figure class="centre">
<img alt="Five infill coasters, printed with different materials and patterns" src="/assets/images/3dp-infill-coasters.jpg" />
</figure>
<h3 id="gravity-hook">Gravity hook</h3>
<p>User <em>Snorri</em> has uploaded designs for a <a href="https://www.thingiverse.com/thing:1607199">gravity hook</a>, which is similar to a grappling hook but has a jaw that opens when the hook rests on the ground and can let you pick up objects.</p>
<p>The comments suggest printing at 105% scale to improve the tolerance; even at this scale I still had to file the screw holes to enlarge them for my M3 screws and sand down the flat spots where the parts mate together. When this was done the hook worked reasonably well, but it was still a little flimsy as the PETG jaws tended to flex.</p>
<p><em>Snorri</em> has also uploaded a second version, <a href="https://www.thingiverse.com/thing:1663821">designed at 200% scale</a>; this worked far better and could be assembled into a working hook straight off the print bed with no post-processing. The larger claw size also means you can pick up larger objects — the office got quite sick of me collecting people’s keys, sunglasses, headphones, mice and various cables (especially when they were plugged in).</p>
<figure class="centre">
<img alt="The large and small size gravity hooks" src="/assets/images/3dp-gravity-hook.jpg" />
</figure>
<h3 id="magnetic-screw-boxes">Magnetic screw boxes</h3>
<p>User <em>Darak_Chen</em> has designed and uploaded a very neat <a href="https://www.thingiverse.com/thing:3423114">magnetic screw box</a>. This is a two-part print, with a box base and a lid section. The base consists of a 3x2 grid of storage compartments, and there are spots for press-fitting small round magnets to ensure the lid remains securely attached.</p>
<p>I found that the lid fit on pretty snugly straight off the printer, so the magnets were probably not strictly necessary. I chose to make the grip a bit looser by sanding down the mating surfaces, and then installed the magnets as designed. The lid jumps on with a satisfying snap and requires a moderate amount of force to dislodge, so you can be sure it won’t come apart in your pocket or bag.</p>
<p>I printed one in translucent green PETG and one in silver PLA, and found the PETG print accepted the magnets much more readily and was generally a more suitable material for this print.</p>
<figure class="half centre">
<img alt="PETG screw box, opened with lid and base visible" src="/assets/images/3dp-magnetic-screw-box1.jpg" />
<img alt="PLA screw box, closed" src="/assets/images/3dp-magnetic-screw-box2.jpg" />
</figure>
<h3 id="extruder-flag">Extruder flag</h3>
<p>PrusaPrinters.org has <a href="https://blog.prusaprinters.org/visualize-extruder-motor-movements/">a great article</a> on the use of 3D printed ‘flags’ to visualise the motion of your extruder motor. This can help you identify skipped steps and overtightened idlers… but really I just think it looks cool.</p>
<p>There’s four different designs to choose from in <a href="https://www.thingiverse.com/thing:2759852">this Thingiverse entry</a> by <em>mikolaszuza</em>. I went with the three triangular flags: simple, but elegant.</p>
<figure class="centre">
<img alt="The triangle flag on the Prusa extruder motor" src="/assets/images/3dp-extruder-flags.jpg" />
</figure>
<h3 id="spoon-rest">Spoon rest</h3>
<p>I was sick of leaving greasy utensils either straight on the bench or balanced on a saucepan while cooking, so I printed a couple of different designs of spoon rest. <a href="https://www.thingiverse.com/thing:7146">This square one</a> by <em>cjbissonnette</em> is fairly utilitarian but works well enough; while I also tried <a href="https://www.thingiverse.com/thing:613192">this round one</a> by <em>rema</em> but had issues printing it, even with supports.</p>
<h3 id="and-some-other-stuff">And some other stuff</h3>
<p>I printed user <em>Bjunior</em>’s <a href="https://www.thingiverse.com/thing:3587079">Tactical Zip Tie Handcuff Joiner</a> for a bit of fun and found it worked surprisingly well (and a big thanks to my dad for being a somewhat-willing test subject!). They’re super quick to print, taking only about 10 minutes each at 0.3mm.</p>
<p>User <em>whistleblower</em> has uploaded — you’ll never guess — <a href="https://www.thingiverse.com/thing:2933021">a whistle</a>. This is <strong>surprisingly</strong> effective; it only requires a bit of air from your mouth before it produces an ear-splitting shriek. I’ve been sporting it on my keyring for the last few weeks as a just-in-case measure; a few friends and colleagues were impressed by the performance and size so I’ve printed more since my first test.</p>
<p>My Raspberry Pi was feeling bare, so I also printed a <a href="https://www.thingiverse.com/thing:2975944">case for it and a 4-channel relay</a>, designed by user <em>Keepars</em>. The Raspberry Pi fits very neatly, but I found the screw holes in my relay module don’t line up properly. I think that there’s a lot of 4-channel relay modules available that look similar but have different hole spacings. Oh well, gives me an opportunity to design my own.</p>
<figure class="centre">
<img alt="The Raspberry Pi/4-channel relay case" src="/assets/images/3dp-raspberry-pi-relay-case.jpg" />
</figure>
<h2 id="whats-next">What’s next?</h2>
<p>Having seen how much fun it is to use a 3D printer and the kinds of things that can be done with it, I’m quite interested in getting my own. This would let me run longer print jobs without worrying about preventing other people from using it, and will also allow me to modify the machine and add or change functionality.</p>
<p>In particular, I’m quite keen to try my hand at replacing the extruder with a pen assembly to turn the machine into a plotter; alternatively, a laser module can make it into a laser engraver and cutter; and I could even design a small pneumatic pump assembly somehow to turn it into a pick-and-place machine.</p>
<p>Additionally, I’ve done some basic work into designing my own 3D models. They’re all quite rudimentary, but it was a fun learning opportunity for me so I plan to follow up this post with another entry on that topic (really — sorry about the two-month gap since my last one!).</p>
<p class="notice--info">I’ve updated this post since I originally published it, as I found links to the filaments that we purchased. These are included above. The silver PLA doesn’t have a link, as it came in the kit and we didn’t buy it separately.</p>sigsecRecently, work bought a 3D printer for prototyping and to encourage staff innovation and creativity. It’s been quite fun playing around with it and seeing designs I’ve found and made come to life. This post is going to be a quick recap of what I’ve printed lately.Garmin Connect code exploration2019-05-01T17:07:14+10:002019-05-01T17:07:14+10:00https://blog.sigsec.net/posts/2019/05/garmin-connect-code-exploration<p>I’m a big fan of fitness tech, and one of my favourite devices is my Garmin fēnix-series sports watch. <a href="https://connect.garmin.com/">Garmin’s Connect</a> platform lets you examine your data and activities, but there’s no public API for us to interact with programmatically. Consequently, it’s a perfect opportunity for me to look at it and see what I can pull out.
<!--more--></p>
<h2 id="garmin-connect">Garmin Connect</h2>
<p><a href="https://connect.garmin.com/">Garmin Connect</a> is the companion app and website for Garmin devices that lets you see your activities, step counts and sleep data, compare your performance with your friends, track races and workout plans, and a whole host of other features. It’s the gateway between a user and their data — if you want to see anything recorded by a Garmin device, you need to go through Connect<sup id="fnref:account-export-note"><a href="#fn:account-export-note" class="footnote">1</a></sup>.</p>
<figure class="half centre">
<img alt="A marketing image of Connect from Garmin" src="/assets/images/garmin-connect-marketing-image.jpg" />
</figure>
<p>While navigating around Connect, it becomes immediately apparent that only a small section of the page needs to update each time — we’re not completely reloading everything. Connect is clearly able to request updates from some server and get back the information it needs. This is typical of modern websites, where there is a user interface (UI) web server that provides all the nice graphical pieces, and a separate back-end API server that provides the raw data to fill in the gaps.</p>
<p>If I can figure out how to talk to the API server without having to use the UI server, then I should be able to replicate this functionality in Python independent of a web browser entirely.</p>
<h2 id="behind-the-scenes">Behind the scenes</h2>
<h3 id="connects-javascript">Connect’s JavaScript</h3>
<p>From opening the developer tools in Chrome, it’s quite apparent that Connect uses technologies like webpack, babel and JSX with the React framework. Projects like this typically get compacted and ‘minified’ to reduce the size of the code the browser needs to load. However, this tends to make debugging nearly impossible as all comments are removed and functions get renamed and rearranged.</p>
<p>Fortunately for us, Connect also provides the ‘source maps’ that contain the file names and source code for the original project. We can attempt to reconstruct the project by processing these source maps using some basic Python code and writing files out to the hard disk.</p>
<p class="notice--info">If you’re interested in reading more about the technologies used to deliver JavaScript on the internet, the Mozilla Developer Network has <a href="https://developer.mozilla.org/en-US/docs/Tools/Debugger/How_to/Use_a_source_map">an excellent writeup</a>.</p>
<h3 id="chunks-and-source-maps">Chunks and source maps</h3>
<p>Some research online lead me to believe that most projects of this sort only have a one compiled JavaScript file with one source map. However Connect has hundreds, split up into ‘chunks’ named like <code class="highlighter-rouge">5.6d2c3e3c.chunk.js</code> or <code class="highlighter-rouge">152.43dceef1.chunk.js</code>. This is an optimisation tactic used to make the page load faster for viewers, as the browser only needs to download the specific chunks it needs at any point in time and request additional ones later, rather than trying to get everything all at the very start.</p>
<p>To reconstruct the source code for the project, we need to be able to pull <em>all</em> of the chunks. This is complicated by the fact that they’re not named strictly numerically — there’s a hash code in there too. Thankfully, the <a href="https://connect.garmin.com/web-react/static/js/bundle.js">base JavaScript file</a> contains a dictionary of all of the chunks that we can use to download them ourselves. Once we have the list of chunks, downloading the source maps is just as easy — just stick <code class="highlighter-rouge">.map</code> at the end of the URL.</p>
<h2 id="recreating-the-sources">Recreating the sources</h2>
<h3 id="getting-the-chunk-list">Getting the chunk list</h3>
<p>I wrote some quick-and-dirty Python code to grab the main JavaScript file and pull out the list of chunk URLs.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">re</span><span class="p">,</span> <span class="n">requests</span>
<span class="n">BASE_URL</span> <span class="o">=</span> <span class="s">"https://connect.garmin.com/web-react/static/js/"</span>
<span class="n">BUNDLE_URL</span> <span class="o">=</span> <span class="n">BASE_URL</span> <span class="o">+</span> <span class="s">"bundle.js"</span>
<span class="k">def</span> <span class="nf">load_chunk_list</span><span class="p">():</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">BUNDLE_URL</span><span class="p">)</span>
<span class="n">chunk_map_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s">r'c\.src=.+?{(0:".+?")}\[e]'</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="n">chunk_map</span> <span class="o">=</span> <span class="n">chunk_map_re</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">','</span><span class="p">)</span>
<span class="n">chunk_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">chunk_map</span><span class="p">:</span>
<span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">":"</span><span class="p">)</span>
<span class="n">chunk_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">s.</span><span class="si">%</span><span class="s">s.chunk.js'</span> <span class="o">%</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="k">return</span> <span class="n">chunk_list</span>
</code></pre></div></div>
<h3 id="downloading-the-source-maps">Downloading the source maps</h3>
<p>From that list, we can then download the source map for each chunk:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">download_all_maps</span><span class="p">(</span><span class="n">chunk_list</span><span class="p">):</span>
<span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">chunk_list</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">BASE_URL</span> <span class="o">+</span> <span class="n">chunk</span> <span class="o">+</span> <span class="s">'.map'</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'chunks/'</span> <span class="o">+</span> <span class="n">chunk</span><span class="p">,</span> <span class="s">'x'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="rehydrating-the-sources">Rehydrating the sources</h3>
<p>Now that we’ve got the source map files, we can go through them to pull out the original code. A source map is a fairly basic JSON file containing an object with a few keys; we’re interested in <code class="highlighter-rouge">sources</code> and <code class="highlighter-rouge">sourcesContent</code>.</p>
<p>My code is pretty messy here so I’ll try and tidy it up for a future post, but the logic is essentially:</p>
<ul>
<li>Go through the <code class="highlighter-rouge">sources</code> and <code class="highlighter-rouge">sourcesContent</code> arrays, element-by-element</li>
<li>Write the <code class="highlighter-rouge">sourcesContent[i]</code> string to a file on the disk named <code class="highlighter-rouge">sources[i]</code></li>
</ul>
<p>There is some manipulation I’m applying to the filename; this is mostly around the fact that some filenames are of the form <code class="highlighter-rouge">webpack:///./node_modules/ui-connect/Notification/Notification.css?e097</code> which can’t make a valid filename.</p>
<h2 id="interesting-finds">Interesting finds</h2>
<p>There’s a lot of interesting information here that we can search through, using a tool like <code class="highlighter-rouge">grep</code> or the ‘find in files’ feature of Sublime Text. Some of the more exciting bits I found were URLs for various API endpoints, as well as comments describing bugs found and fixed in the past.</p>
<h3 id="future-devices-and-connect-features">Future devices and Connect features</h3>
<p class="notice--warning">I wrote this post a few days ago but held off on publishing it while I tidied up the code. Evidently I lost my opportunity to break the news because the <a href="https://buy.garmin.com/en-US/US/p/621922/pn/010-02063-00">Forerunner 945</a> and <a href="https://www.garmin.com/en-US/blog/featured-2/menstrual-cycle-tracking-in-garmin-connect/">Menstrual Cycle Tracking</a> features are now publicly announced.</p>
<p>There’s code and UI designs for what appears to be an unannounced new feature — Menstrual Cycle tracking. This must be nearly ready for release because it’s in the publicly-available version of Garmin Connect and is only protected by an ‘is this user a beta-tester’ flag. While this doesn’t directly impact me, it’s cool to see Garmin working on getting feature parity with something that Fitbit have had <a href="https://blog.fitbit.com/fitbit-period-tracker/">for some time</a>.</p>
<p>There are also reference to a future device — The Forerunner 945 (called <code class="highlighter-rouge">FR 945</code> in the code). Some research online shows that Garmin haven’t <em>officially</em> announced it yet, but it has been <a href="https://the5krunner.com/2019/04/23/garmin-forerunner-945-specs-leaked/">extensively leaked</a> and estimates are pegging the reveal date as any time in the next few days.</p>
<h3 id="other-stuff">Other stuff</h3>
<p>There’s a lot of other interesting bits and pieces in here that I’ll try to take a look at in a future post, especially considering I haven’t even scratched the surface of interacting with the API yet.</p>
<p>Perhaps next time I’ll also spend less time on polish and look at getting the first mover advantage; it would be cool (even if somewhat vain) to see my blog cited as the source for a big feature announcement like this.</p>
<div class="footnotes">
<ol>
<li id="fn:account-export-note">
<p>It is also possible to <a href="https://www.garmin.com/en-US/account/datamanagement/exportdata/">request an export</a> of your Garmin account, but this is an all-or-nothing approach that can “take up to 30 days” to process. <a href="#fnref:account-export-note" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>sigsecI’m a big fan of fitness tech, and one of my favourite devices is my Garmin fēnix-series sports watch. Garmin’s Connect platform lets you examine your data and activities, but there’s no public API for us to interact with programmatically. Consequently, it’s a perfect opportunity for me to look at it and see what I can pull out.TLS with S3 static websites2019-04-23T23:10:26+10:002019-04-23T23:10:26+10:00https://blog.sigsec.net/posts/2019/04/tls-with-s3-static-websites<p>It’s been quite a while since I <a href="/posts/2018/09/aws-and-jekyll.html">set up my blog</a> using AWS S3’s static website hosting and while it’s been great so far, there’s one long-overdue feature I’ve been missing: transport encryption. I’m ashamed it’s taken me this long to add it, but it’s now available and you can learn how to set it up yourself!
<!--more--></p>
<p>But perhaps most importantly, you can see all the problems I faced and avoid all the mistakes I made.</p>
<h2 id="architecture">Architecture</h2>
<p>Up to now, I’ve just been serving HTTP requests directly from an Amazon S3 bucket that’s been configured as a static web host. However, S3 static websites aren’t able to support HTTPS. To get HTTPS working, I could set up a reverse proxy such as <a href="https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/">nginx</a> to handle this, but instead I’m going to use <a href="https://aws.amazon.com/cloudfront/">Amazon CloudFront</a>.</p>
<p>One of the benefits of using CloudFront is that it can cache static resources for us ‘at the edge’, using up to 169 different physical endpoints (depending on the availability tier selected). This will greatly improve the performance of my website for viewers around the world.</p>
<p>The only missing piece here is the TLS certificate we’ll use to do the encryption in CloudFront, and verify that we own and control our domain name. We’re able to take care of this using AWS Certificate Manager.</p>
<h2 id="setting-it-up">Setting it up</h2>
<h3 id="tls-certificate">TLS certificate</h3>
<ol>
<li>
<p>Log into the AWS console and navigate to the <a href="https://console.aws.amazon.com/acm/home?region=us-east-1">AWS Certificate Manager</a> admin panel. If you’re like me and have never used Certificate Manager before, you’ll see a welcome screen with a couple of icons.</p>
<p class="notice--warning"><strong>Note:</strong> You <em>must</em> change your console region to <code class="highlighter-rouge">us-east-1</code> (US East, N. Virginia), or the certificates we generate <a href="https://docs.aws.amazon.com/acm/latest/userguide/acm-regions.html">won’t work in CloudFront</a>.</p>
</li>
<li>Under the ‘Provision certificates’ icon click <em>Get started</em>.</li>
<li>Select <em>Request a public certificate</em> from the two options shown, then click <em>Request a certificate</em>.</li>
<li>
<p>You’ll be asked to enter the fully-qualified domain names (FQDNs) that will be on the certificate, such as <code class="highlighter-rouge">contoso.net</code> and <code class="highlighter-rouge">blog.contoso.net</code>.</p>
<p class="notice--info"><strong>Tip:</strong> You can use a single asterisk to match ‘any name’, so if you have a bunch of subdomains you can match all of them (and any future additions) by using <code class="highlighter-rouge">*.contoso.net</code>.</p>
<figure>
<img alt="The domain settings used to create our certificate" src="/assets/images/blog-tls-certificate-domains.png" />
</figure>
</li>
<li>When you’ve added your domain(s), click <em>Next</em>.</li>
<li>You’ll be asked whether you want to use DNS validation or email validation to prove you own or control the domains listed. I chose <strong>DNS validation</strong>. Click <em>Review</em>.</li>
<li>Amazon will provide a summary page listing the domain names you want to secure and your validation method. Click <em>Confirm and request</em> if you’re happy with the info.</li>
<li>
<p>Amazon will think for a few moments, before providing <code class="highlighter-rouge">CNAME</code> records that must be set on the domains you have provided. Depending on your DNS provider, the instructions to do this will vary.</p>
<p class="notice--info"><strong>Tip:</strong> While you can technically delete the <code class="highlighter-rouge">CNAME</code> records after the certificate is issued, leaving them there means Amazon can <a href="https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-validate-dns.html#dns-delete-cname">automatically renew</a> your certificates when they come close to expiry.</p>
</li>
</ol>
<p>That’s it! It will take a little while for Amazon to discover your new <code class="highlighter-rouge">CNAME</code> record; it took about five minutes for me.</p>
<h3 id="cloudfront">CloudFront</h3>
<p>Now we have our encryption certificate, we can set up CloudFront.</p>
<ol>
<li>Log into the AWS Console and navigate to the <a href="https://console.aws.amazon.com/cloudfront/home">CloudFront admin panel</a>.</li>
<li>I haven’t used CloudFront before, so there was a message telling me to set up a distribution. Just click the big blue <em>Create Distribution</em> button.</li>
<li>We’re serving web files, so click <em>Get Started</em> under the ‘Web’ heading.</li>
<li>Now we need to fill in a bunch of form fields.
<ol>
<li><strong>Origin Domain Name</strong>: This is the full URL of our bucket (such as <code class="highlighter-rouge">blog.contoso.net.s3-website-ap-southeast-2.amazonaws.com</code>). It should create an auto-fill dropdown when you click in the box.</li>
<li><strong>Viewer Protocol Policy</strong>: I picked <code class="highlighter-rouge">Redirect HTTP to HTTPS</code>.</li>
<li><strong>Compress Objects Automatically</strong>: I picked <code class="highlighter-rouge">Yes</code>. This will reduce network overhead, should make pages load quicker for viewers, and is <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html">less expensive for you</a> (as costs are incurred based on the quantity of data transferred).</li>
<li><strong>Price Class</strong>: I went with <code class="highlighter-rouge">Use Only U.S., Canada and Europe</code> as it’s the cheapest tier and I’m not too concerned about latency.</li>
<li><strong>Alternate Domain Names (CNAMEs)</strong>: Put in your domain names here (for my example, <code class="highlighter-rouge">blog.contoso.net</code>).</li>
<li><strong>SSL Certificate</strong>: Pick <em>Custom SSL certificate</em> and use the one created in the previous heading.</li>
<li><strong>Default Root Object</strong>: I put <code class="highlighter-rouge">index.html</code>; if you leave this out, you may get either a <code class="highlighter-rouge">ListBucketResult</code> (like I did) or an <code class="highlighter-rouge">AccessDenied</code> error.</li>
<li>The other settings are fine to be left at their defaults.</li>
</ol>
</li>
<li>Click <em>Create Distribution</em>.</li>
</ol>
<p>Another step done! It will take a while for for CloudFront to mirror your S3 bucket to its edge nodes. In my case it took about 40 minutes, but I was playing with settings while it was working so it’s possible this is longer than usual.</p>
<h3 id="updating-the-domain">Updating the domain</h3>
<p>At this point, we have a shiny new TLS certificate to verify that we own our domain and encrypt traffic to and from it; and we have our website mirrored in CloudFront in a bunch of places around the world.</p>
<p class="notice--info"><strong>Tip:</strong> If you’re following these steps for yourself, I’d recommend leaving this heading until after you’ve fixed the bugs described in the headings below.</p>
<p>The only thing left to do is point our <code class="highlighter-rouge">blog.contoso.net</code> domain at the CloudFront address. This is a very similar process to adding the <code class="highlighter-rouge">CNAME</code> record in the first heading; this time, we’re using the <em>Domain Name</em> that CloudFront provides us and putting it in as another <code class="highlighter-rouge">CNAME</code> record for our subdomain.</p>
<p>If you followed my <a href="/posts/2018/09/aws-and-jekyll.html">earlier article</a> you’ll already have a <code class="highlighter-rouge">CNAME</code> set up for this purpose, we can just change the value from the <code class="highlighter-rouge">s3-website</code> address into our <code class="highlighter-rouge">cloudfront.net</code> address.</p>
<p>You can be sure it’s working when you see that glorious little green padlock in the URL bar when you visit your domain:</p>
<figure class="third centre">
<img alt="The security padlock in the URL bar for my site" src="/assets/images/blog-tls-domain-padlock.png" />
</figure>
<h2 id="oh-no-theres-bugs">Oh no, there’s bugs</h2>
<p>At this point, I <em>thought</em> my website behaved exactly the same way as before, just with HTTPS. However, I quickly ran into a couple of issues:</p>
<ul>
<li>404 pages are broken. I’ve set up a custom 404 handler page in S3’s static website hosting option, but this isn’t respected by CloudFront. These errors now just give you a cryptic <code class="highlighter-rouge">AccessDenied</code> error in XML.</li>
<li>Navigating to a folder (eg <code class="highlighter-rouge">/categories/</code>) doesn’t automatically fetch the <code class="highlighter-rouge">index.html</code> file inside it. It instead gives you the same error as in the previous dot point.</li>
</ul>
<p>Some research indicated the quick-and-dirty solution was to <a href="https://stackoverflow.com/a/15528757">change the origin</a> for the CloudFront distribution from the internal S3 bucket reference, to the external S3 static-website URL. This gives you all the benefits of S3’s URL rewriting (for folder indexes) and 404-page handling, with all the benefits of CloudFront’s distribution network and HTTPS. Sounds perfect, right?</p>
<p>Except there’s a catch — CloudFront will serve HTTPS to your viewers, but it’s actually fetching content from your S3 bucket <strong>over unencrypted HTTP</strong>! This might not be a big deal for some of you, but I’d like my traffic to be encrypted end-to-end (and also remove direct public access to the bucket).</p>
<h3 id="fixing-http-errors">Fixing HTTP errors</h3>
<p>We can resolve a lot of the HTTP errors by creating a ‘Custom Error Response’ in our CloudFront distribution.</p>
<ol>
<li>Log into CloudFront and click on the ID of your distribution in the list.</li>
<li>Click on the <em>Error Pages</em> tab, then click <em>Create Custom Error Response</em>.</li>
<li>In the ‘HTTP Error Code’ box, pick <code class="highlighter-rouge">403: Forbidden</code> and set ‘Customize Error Response’ to <em>Yes</em>.</li>
<li>In the ‘Response Page Path’ box, put in <code class="highlighter-rouge">/404.html</code> (or whatever your static 404 page is named). Select <code class="highlighter-rouge">404: Not Found</code> in the ‘HTTP Response Code’ box.</li>
<li>Click <em>Create</em>.</li>
</ol>
<p class="notice--info"><strong>Tip:</strong> While we’re <em>serving</em> a 404 page to our visitors, CloudFront internally <em>receives</em> a 403 error from S3. This is why we set ‘HTTP Error Code’ to 403 instead of 404.</p>
<h3 id="fixing-directory-indexes">Fixing directory indexes</h3>
<p>There’s two ways we can fix directory <code class="highlighter-rouge">index.html</code> files not being served. We can either:</p>
<ul>
<li>Rename folder indexes like <code class="highlighter-rouge">/categories/index.html</code> to the extension-less file <code class="highlighter-rouge">/categories</code>; or</li>
<li>Use Lambda@Edge to rewrite requests for directories into their <code class="highlighter-rouge">index.html</code> files.</li>
</ul>
<p>Renaming the S3 records is cheap but involves a post-processing step that can clutter up your bucket (depending on your workflow); while Lambda@Edge can increase your costs depending on how many times it needs to run.</p>
<p>I chose to play around with Lambda@Edge, based on a <a href="https://aws.amazon.com/blogs/compute/implementing-default-directory-indexes-in-amazon-s3-backed-amazon-cloudfront-origins-using-lambdaedge/">blog post from Amazon</a>.</p>
<ol>
<li>
<p>Once you’ve created your CloudFront distribution, you’ll need to go to the <a href="https://console.aws.amazon.com/lambda/home">Lambda console</a>.</p>
<p class="notice--warning"><strong>Note:</strong> As before, you’ll need to change your console region to <code class="highlighter-rouge">us-east-1</code> (US East, N. Virginia).</p>
</li>
<li>Click the orange <em>Create function</em> button.
<ol>
<li>You should see a row of three options above a form; ensure <em>Author from scratch</em> is selected.</li>
<li>In the form, enter a function name and choose <code class="highlighter-rouge">Node.js 8.10</code> for the <em>Runtime</em>. I chose to name my function <code class="highlighter-rouge">CloudFront-S3-index-files</code>.</li>
</ol>
</li>
<li>Expand the <em>Choose or create an execution role</em> link.
<ol>
<li>Set <em>Execution role</em> to <code class="highlighter-rouge">Create a new role from AWS policy templates</code>.</li>
<li>Enter a <em>Role name</em> similar to your <em>Function name</em>. I chose <code class="highlighter-rouge">CloudFront-S3-index-files-role</code>.</li>
<li>In the <em>Policy templates</em> box, select <code class="highlighter-rouge">Basic Lambda@Edge permissions</code>.</li>
</ol>
</li>
<li>Click <em>Create function</em>. Amazon will think for a few moments, before dropping you into the Lambda editor.</li>
<li>
<p>In the editor at the bottom, copy and paste the following code:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">'</span><span class="s1">use strict</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">handler</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">request</span> <span class="o">=</span> <span class="nx">event</span><span class="p">.</span><span class="nx">Records</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">cf</span><span class="p">.</span><span class="nx">request</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">oldUri</span> <span class="o">=</span> <span class="nx">request</span><span class="p">.</span><span class="nx">uri</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">newUri</span> <span class="o">=</span> <span class="nx">oldUri</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/</span><span class="se">\/</span><span class="sr">$/</span><span class="p">,</span> <span class="dl">'</span><span class="se">\</span><span class="s1">/index.html</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Old URI: </span><span class="dl">"</span> <span class="o">+</span> <span class="nx">oldUri</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">New URI: </span><span class="dl">"</span> <span class="o">+</span> <span class="nx">newUri</span><span class="p">);</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">uri</span> <span class="o">=</span> <span class="nx">newUri</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">callback</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="nx">request</span><span class="p">);</span>
<span class="p">};</span>
</code></pre></div> </div>
</li>
<li>Click the orange <em>Save</em> button in the header.</li>
<li>Looking at the ‘Designer’ part (top half) of the form, click <em>CloudFront</em> in the menu on the left.</li>
<li>The bottom half of the form will ask you to configure the trigger — click <em>Deploy to Lambda@Edge</em>. A modal dialog will pop up, asking for some more information.
<ol>
<li><strong>Distribution</strong>: Pick the ID of the CloudFront distribution we created earlier</li>
<li><strong>Cache behavior</strong>: <code class="highlighter-rouge">*</code></li>
<li><strong>CloudFront event</strong>: <code class="highlighter-rouge">Origin request</code></li>
<li>Check the <code class="highlighter-rouge">I acknowledge ...</code> box.</li>
<li>Click <em>Deploy</em>. The modal will close and you’ll be returned to the editor.</li>
</ol>
<figure class="half centre">
<img alt="The Lambda@Edge CloudFront trigger properties" src="/assets/images/blog-tls-lambda-trigger.png" />
</figure>
</li>
</ol>
<p>You’re done! The Lambda should be effective immediately, and you can test it by visiting one of your directory pages (like <code class="highlighter-rouge">/categories/</code> in my case).</p>
<h2 id="wrap-up">Wrap up</h2>
<p>Wow, we’re <em>finally</em> done. Our site is now neatly protected by TLS, we have the added benefit of improved performance for a fair chunk of the world, and we resolved a bunch of confusing and time-consuming problems we encountered along the way.</p>
<p>The one final issue you might encounter from here on out is around ‘invalidating’ cache content: if you delete or modify a file in your bucket, the CloudFront nodes might not necessarily pick up on this change straight away.</p>
<p>We can tell CloudFront to invalidate files manually in a few different ways; if, like me, you’re using <code class="highlighter-rouge">s3cmd sync</code> to push your site into a bucket, the easy option is tacking on a <code class="highlighter-rouge">--cf-invalidate</code> parameter<sup id="fnref:s3cmd-error"><a href="#fn:s3cmd-error" class="footnote">1</a></sup>. This will handle cache invalidate for any files changed by the sync operation (you might need to add some additional policies to the IAM user you’re using to sync).</p>
<div class="footnotes">
<ol>
<li id="fn:s3cmd-error">
<p>There is <a href="https://github.com/s3tools/s3cmd/issues/1006">currently a bug</a> in the latest release, but it has been fixed awaiting the next version. <a href="#fnref:s3cmd-error" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>sigsecIt’s been quite a while since I set up my blog using AWS S3’s static website hosting and while it’s been great so far, there’s one long-overdue feature I’ve been missing: transport encryption. I’m ashamed it’s taken me this long to add it, but it’s now available and you can learn how to set it up yourself!Reading and writing NFC tags2019-04-08T09:10:40+10:002019-04-08T09:10:40+10:00https://blog.sigsec.net/posts/2019/04/reading-and-writing-nfc-tags<p>Now that we’ve resolved all of the major <a href="/posts/2019/04/nfc-fun-with-python.html">teething issues we had</a> with Python 3 and nfcpy, we can look at reading the contents of tags presented by the user and writing our own. There’s a wide variety of tags that can be written, so I’ll cover a couple of the most interesting — Wi-Fi credentials and URIs.
<!--more--></p>
<h2 id="structural-nfc-code">Structural NFC code</h2>
<p>The structural code here is a bit involved, so I’ll break it up into sections and explain each bit.</p>
<p>The first step is importing modules and opening a handle to our <code class="highlighter-rouge">ContactlessFrontend</code> — the device we’re using as the interface between the computer running Python and our NFC devices in the real world. In my case, I’m using a USB-connected ACR122U device; if you’re not sure where <code class="highlighter-rouge">usb:072f:2200</code> comes from, read the <a href="/posts/2019/04/nfc-fun-with-python.html">first part</a> of the series.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">sleep</span>
<span class="kn">import</span> <span class="nn">nfc</span>
<span class="k">with</span> <span class="n">nfc</span><span class="o">.</span><span class="n">ContactlessFrontend</span><span class="p">(</span><span class="s">'usb:072f:2200'</span><span class="p">)</span> <span class="k">as</span> <span class="n">clf</span><span class="p">:</span>
</code></pre></div></div>
<p>Once the frontend is opened, we need to sense for tags. However, we don’t just want to check for tags once and bail out; so I stick the code in a <code class="highlighter-rouge">while True</code> loop to handle endlessly looking for tags.</p>
<p>For this example, I’m only looking for <code class="highlighter-rouge">106A</code>-class tags, but there are a couple others we could look for too. If we don’t find anything (<code class="highlighter-rouge">target</code> equals <code class="highlighter-rouge">None</code>), we wait for 100ms to stop the CPU from burning up, and then search again.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">target</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">sense</span><span class="p">(</span><span class="n">RemoteTarget</span><span class="p">(</span><span class="s">'106A'</span><span class="p">))</span>
<span class="k">if</span> <span class="n">target</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span> <span class="c1"># don't burn the CPU
</span> <span class="k">continue</span>
<span class="n">serial</span> <span class="o">=</span> <span class="n">target</span><span class="o">.</span><span class="n">sdd_res</span><span class="o">.</span><span class="nb">hex</span><span class="p">()</span>
</code></pre></div></div>
<p>At this point, we’ve determined that there <em>is</em> an NFC target near the reader and we also have its serial number. We could even bail out here if checking the serial number was sufficient for us. The last step in getting the juicy information out (like NDEF records contained) is to <code class="highlighter-rouge">activate</code> it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">tag</span> <span class="o">=</span> <span class="n">nfc</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">activate</span><span class="p">(</span><span class="n">clf</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
</code></pre></div></div>
<p>At this point, we’re ready to move onto something more interesting. For example, we might want to read the contents of the records, run a query against a database, or write something back.</p>
<h2 id="reading-and-writing">Reading and writing</h2>
<h3 id="reading-records">Reading records</h3>
<p>Now that we’ve found an NFC device near us and have activated it, we can start to pull information out. The <code class="highlighter-rouge">ndef</code> property of our tag object lets us access the NDEF memory area on the chip; this contains all of the most exciting information. Depending on what we’re expecting to find and what we get, we might need to take various different actions.</p>
<p>For the purposes of this example, I’ll just print out the record information found. You’ll need to modify the code to behave differently for your specific situation.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">if</span> <span class="ow">not</span> <span class="n">tag</span><span class="o">.</span><span class="n">ndef</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"No NDEF records found!"</span><span class="p">)</span>
<span class="k">continue</span>
<span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">tag</span><span class="o">.</span><span class="n">ndef</span><span class="o">.</span><span class="n">records</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Found record: "</span> <span class="o">+</span> <span class="n">record</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="writing-records">Writing records</h3>
<p>It’s easy to write NDEF records to a card — reassign the <code class="highlighter-rouge">tag.ndef.records</code> field:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">ndef</span>
<span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">TextRecord</span><span class="p">(</span><span class="s">"Hello sigsec readers!"</span><span class="p">)</span>
<span class="c1"># 'tag' is an nfcpy tag acquired earlier...
</span><span class="n">tag</span><span class="o">.</span><span class="n">ndef</span><span class="o">.</span><span class="n">records</span> <span class="o">=</span> <span class="p">[</span><span class="n">record</span><span class="p">]</span>
</code></pre></div></div>
<p>This does require that the card is <strong>already NDEF formatted</strong>; it’s possible you might get a card that doesn’t come adequately prepared, so the code may fail in this case. For more info and some background reading, head to the <a href="https://nfcpy.readthedocs.io/en/latest/topics/get-started.html#read-and-write-tags">nfcpy documentation</a>.</p>
<p class="notice--warning"><strong>Note:</strong> The <code class="highlighter-rouge">nfc.ndef</code> package is deprecated and will cause problems if used. Stick to the <code class="highlighter-rouge">ndef</code> package in the <code class="highlighter-rouge">ndeflib</code> pip module.</p>
<h2 id="ndef-record-types">NDEF record types</h2>
<p>There are a number of different types of records that we can create, with different uses and applications. To write any of these to a smartcard or NFC tag, use the steps in the heading above with the <code class="highlighter-rouge">record</code> object produced below.</p>
<h3 id="text-records">Text records</h3>
<p>Text records are super easy; they contain a string of some text, a language key and an encoding. If you don’t specify anything, <code class="highlighter-rouge">ndeflib</code> will use <code class="highlighter-rouge">en</code> for the language and <code class="highlighter-rouge">UTF-8</code> for the encoding by default.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">TextRecord</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">language</span><span class="p">,</span> <span class="n">encoding</span><span class="p">)</span>
<span class="c1"># eg. record = ndef.TextRecord("Hello sigsec readers!")
</span></code></pre></div></div>
<h3 id="wi-fi-config-records">Wi-Fi config records</h3>
<p>Making a Wi-Fi configuration record is a little bit more involved due to the number of parameters we need to set. Once the parameters are identified, it’s just as straight-forward. If you get stuck, try reading the <code class="highlighter-rouge">ndeflib</code> <a href="https://ndeflib.readthedocs.io/en/stable/records/wifi.html#configuration-token">documentation for this feature</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">credential</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">wifi</span><span class="o">.</span><span class="n">Credential</span><span class="p">()</span>
<span class="n">credential</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'ssid'</span><span class="p">,</span> <span class="n">b</span><span class="s">'MyNetworkSSID'</span><span class="p">)</span>
<span class="n">credential</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'authentication-type'</span><span class="p">,</span> <span class="s">'WPA2-Personal'</span><span class="p">)</span>
<span class="n">credential</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'encryption-type'</span><span class="p">,</span> <span class="s">'AES'</span><span class="p">)</span>
<span class="n">credential</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'network-key'</span><span class="p">,</span> <span class="n">b</span><span class="s">'WiFi-Pa$$w0rd'</span><span class="p">)</span>
<span class="n">credential</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'mac-address'</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">([</span><span class="mh">0xFF</span><span class="p">,</span> <span class="mh">0xFF</span><span class="p">,</span> <span class="mh">0xFF</span><span class="p">,</span> <span class="mh">0xFF</span><span class="p">,</span> <span class="mh">0xFF</span><span class="p">,</span> <span class="mh">0xFF</span><span class="p">]))</span>
<span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">WifiSimpleConfigRecord</span><span class="p">()</span>
<span class="n">record</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'credential'</span><span class="p">,</span> <span class="n">credential</span><span class="p">)</span>
<span class="n">record</span><span class="o">.</span><span class="n">set_attribute</span><span class="p">(</span><span class="s">'rf-bands'</span><span class="p">,</span> <span class="p">(</span><span class="s">'2.4GHz'</span><span class="p">,</span> <span class="s">'5.0GHz'</span><span class="p">))</span>
</code></pre></div></div>
<h3 id="uri-records">URI records</h3>
<p>A lot of useful actions can be constructed using the URI record. The basic form of the record constructor is:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">UriRecord</span><span class="p">(</span><span class="s">"https://www.example.org/"</span><span class="p">)</span>
</code></pre></div></div>
<p>If you want to go straight to Google Maps navigation, put in an address in this format (or look here for <a href="https://developers.google.com/maps/documentation/urls/android-intents">more Google intents</a>):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">UriRecord</span><span class="p">(</span><span class="s">"google.navigation:q=1600+Pennsylvania+Avenue,+Washington+DC"</span><span class="p">)</span>
</code></pre></div></div>
<p>To compose an email (even including a pre-filled subject line), use this format:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">UriRecord</span><span class="p">(</span><span class="s">"mailto:recipient@example.com?subject=Hello</span><span class="si">%20</span><span class="s">there"</span><span class="p">)</span>
</code></pre></div></div>
<p>To dial a phone number, use this format:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">record</span> <span class="o">=</span> <span class="n">ndef</span><span class="o">.</span><span class="n">UriRecord</span><span class="p">(</span><span class="s">"tel:+61400123456"</span><span class="p">)</span>
</code></pre></div></div>
<p>There’s a lot of other actions that can happen: if you’ve installed apps like YouTube, Facebook or Twitter, putting in links to videos or profiles in these services will generally cause the corresponding app to open automatically when the tag is read. This can be a great way to get people in touch with you if you want to put NFC stickers onto business cards or belongings.</p>
<h2 id="whats-next">What’s next?</h2>
<p>For me, I’ve ordered a set of 10 NFC stickers off eBay to play around with. I think I’ll write Wi-Fi configurations onto a couple of them for use at home and in the office. With the rest, though, I don’t have any specific plan in mind.</p>
<p>I like the idea of connecting the reader to a database so I can handle basic scanning in and out of belongings, like if I loan something to a friend… but this seems like it could be more easily done with just a spreadsheet and manual tracking.</p>
<p>Projects I’ve seen online from others include making access control systems, time clocks, inventory management and a lot of other security related applications. Perhaps in the future I’ll write a post about one of these uses.</p>sigsecNow that we’ve resolved all of the major teething issues we had with Python 3 and nfcpy, we can look at reading the contents of tags presented by the user and writing our own. There’s a wide variety of tags that can be written, so I’ll cover a couple of the most interesting — Wi-Fi credentials and URIs.NFC fun with Python2019-04-01T08:48:33+11:002019-04-04T23:14:58+11:00https://blog.sigsec.net/posts/2019/04/nfc-fun-with-python<p>I’ve recently been looking into using an NFC reader/writer pad at work to make configuring some devices we have much quicker and easier, as the standard process for these devices is an Android phone and a lot of manual data entry. Checking the device using a mobile is great, but configuring them this way is not scalable.
<!--more--></p>
<h2 id="setup">Setup</h2>
<h3 id="the-hardware">The hardware</h3>
<p>I did some research and found we could get an NFC pad and write some Python code using <a href="https://github.com/nfcpy/nfcpy">nfcpy</a> to automate much of the labour, reducing the time taken and decreasing the likelihood of human error. Some quick searches on eBay showed that the <a href="https://www.acs.com.hk/en/products/3/acr122u-usb-nfc-reader/">ACS ACR122U</a> was a reasonably cost-effective option, so I ordered a couple of units for testing.<sup id="fnref:image-source"><a href="#fn:image-source" class="footnote">1</a></sup></p>
<figure class="third centre">
<img alt="The ACR122U hardware" src="/assets/images/nfc-acr122u.png" />
</figure>
<p>When they arrived, I unpacked one and powered it up — so far, so good. Linux seems to have a basic driver for it that makes it beep when you bring a card near (at least that’s all I could get it to do). It’s also a fairly powerful reader — mine was able to communicate with test tags up to 50mm away, including through 25mm particleboard.</p>
<p>The next step is getting the reader working with Python.</p>
<h3 id="using-the-reader-with-nfcpy">Using the reader with nfcpy</h3>
<p>At this point, I immediately encountered problems. The most recent version of nfcpy <a href="https://github.com/nfcpy/nfcpy/issues/47"><em>does not support Python 3</em></a>. In particular, there a number of places where it treats <code class="highlighter-rouge">str</code> and <code class="highlighter-rouge">bytes</code> types the same, which is no longer the case. My options were to move back to Python 2, port the module to Python 3 or abandon the idea entirely — I decided to try and port the library to Python 3. I’ve spent a fair amount of time on it so it can work for my device; feel free to grab the code from <a href="https://github.com/sigsec/nfcpy/">my fork</a>, but be aware it might not work for you out-of-the-box.</p>
<p>Part way through this process, I also discovered that nfcpy has <a href="https://github.com/nfcpy/nfcpy/blob/master/docs/overview.rst#general-notes">this to say</a> of the ACR122U:</p>
<blockquote>
<p>It can not be overstated that the ACR122U is not a good choice for <em>nfcpy</em>.</p>
</blockquote>
<p>So, yeah. The lesson for me here is <strong>do (better) research before I commit to a plan</strong>. I hope you’ll learn from my mistake.</p>
<h3 id="linux-device-access-issues">Linux device access issues</h3>
<p>While porting the code, I also hit upon the issue of the device being locked out, so Python couldn’t use it. Some research led me to running <code class="highlighter-rouge">python -m nfc</code>, which helpfully reports that it’s a kernel module causing the problems:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** found usb:072f:2200 at usb:001:008 but it's already used
-- scan sysfs entry at '/sys/bus/usb/devices/1-1:1.0/'
-- the device is used by the 'pn533_usb' kernel driver
-- this kernel driver belongs to the linux nfc subsystem
-- you can remove it to free the device for this session
sudo modprobe -r pn533_usb
-- and blacklist the driver to prevent loading next time
sudo sh -c 'echo blacklist pn533_usb >> /etc/modprobe.d/blacklist-nfc.conf'
</code></pre></div></div>
<p>As indicated, a temporary fix is running <code class="highlighter-rouge">sudo modprobe -r pn533_usb</code>; or for a permanent solution blacklist the module:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo blacklist pn533_usb >> /etc/modprobe.d/blacklist-nfc.conf'</span>
</code></pre></div></div>
<h2 id="wireshark-usb-analysis">Wireshark USB analysis</h2>
<p>Part way through migrating the module to Python 3, I started encountering issues where the <code class="highlighter-rouge">nfc-poll</code> command on Linux would work fine, but Python was indicating that the NFC chip wasn’t responding with enough data. This seemed a little bit suspect to me, so I decided to crack out <a href="https://www.wireshark.org/">Wireshark</a> to do some USB sniffing and find out where the differences were.</p>
<h3 id="linux-permissions">Linux permissions</h3>
<p>Wireshark can capture data from USB devices using the various <code class="highlighter-rouge">usbmon</code> adapters. Unfortunately the permissions on these are by default pretty locked down on Linux, so some searching led me to <a href="https://superuser.com/q/619473">this helpful SuperUser post</a>. The <em>temporary hack</em> to fix this involves granting everyone read and write access to the file: <code class="highlighter-rouge">sudo chmod o=rw /dev/usbmon1</code>.</p>
<p class="notice--info"><strong>Tip</strong>: If you’re missing <code class="highlighter-rouge">/dev/usbmon*</code> files, chances are you haven’t loaded the kernel module — run <code class="highlighter-rouge">sudo modprobe usbmon</code>. Following the instructions below will take care of this for you automatically from this point onwards.</p>
<p>A more robust solution is to create a <code class="highlighter-rouge">usbmon</code> group and tell the <code class="highlighter-rouge">udev</code> daemon to grant appropriate access. These instructions add you to the newly-created group; you’ll <strong>need to logout</strong> for the group changes to take effect (or you can try your luck with <a href="https://superuser.com/a/609141">these instructions</a>).</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>addgroup usbmon
<span class="nv">$ </span><span class="nb">sudo </span>usermod <span class="nt">-a</span> <span class="nt">-G</span> usbmon <span class="nv">$USER</span>
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'SUBSYSTEM=="usbmon", MODE="640", GROUP="usbmon"'</span> | <span class="nb">sudo tee</span> /etc/udev/rules.d/99-usbmon.rules
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'SUBSYSTEM=="usb", RUN+="/sbin/modprobe usbmon"'</span> | <span class="nb">sudo tee</span> /etc/udev/rules.d/99-usbmon.rules
<span class="nv">$ </span><span class="nb">sudo </span>udevadm trigger
</code></pre></div></div>
<h3 id="opening-a-usb-capture">Opening a USB capture</h3>
<p>You might have a number of <code class="highlighter-rouge">usbmon</code> devices that Wireshark can attach to. To work out which one you need, run <code class="highlighter-rouge">lsusb</code> and look for your device. The Bus number (<code class="highlighter-rouge">001</code>) means that we will use <code class="highlighter-rouge">usbmon1</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Bus 001 Device 020: ID 072f:2200 Advanced Card Systems, Ltd ACR122U
</code></pre></div></div>
<p>Depending on what other devices are on the bus you’re capturing on, you could be flooded by traffic pretty quickly. You can reduce what is shown using a display filter of <code class="highlighter-rouge">usbccid</code> (to jump straight to the display filter box in Wireshark, press <kbd>Ctrl</kbd>+<kbd>/</kbd>).</p>
<h3 id="dissectors-and-filtering">Dissectors and filtering</h3>
<p>Wireshark should now be capturing USB traffic and looking for CCID packets to display. Chances are, there might not be anything there right now. We can get some messages to appear in Wireshark by running the <code class="highlighter-rouge">nfc-poll</code> command in a terminal; it will look for your NFC card reader and interrogate it. To get some informative output, put an NFC card on the top of the reader before you run the command. Here’s output from when I ran it:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nfc-poll uses libnfc 1.7.1
NFC reader: ACS / ACR122U PICC Interface opened
NFC device will poll during 30000 ms (20 pollings of 300 ms for 5 modulations)
ISO/IEC 14443A (106 kbps) target:
ATQA (SENS_RES): 00 04
UID (NFCID1): fc 09 2e 06
SAK (SEL_RES): 08
nfc_initiator_target_is_present: Target Released
Waiting for card removing...done.
</code></pre></div></div>
<p>Once Wireshark has captured some USB traffic, you can take a look through it to try and spot any issues. Unfortunately, it doesn’t tell us much about the content of the messages by default. We can tell Wireshark that we know the messages are from an ACR122U device by changing the decoding.</p>
<p><strong>If you’ve got rows with <code class="highlighter-rouge">USBCCID</code></strong> in the Protocol column, just right click one of them and select ‘Decode As…’ in the context menu. A dialog will open with a table with one row. Change the <code class="highlighter-rouge">(none)</code> in the ‘Current’ column to <code class="highlighter-rouge">ACR 122</code>, then click OK:</p>
<figure class="half centre">
<img alt="Selecting the ACR 122 dissector in Wireshark" src="/assets/images/nfc-wireshark-dissector1.png" />
</figure>
<p>On the other hand, if you <strong>don’t have any <code class="highlighter-rouge">USBCCID</code> rows</strong> we can still get the same result. Just right click one of the USB rows and pick ‘Decode As…’ in the context menu. In the <code class="highlighter-rouge">USB product</code> row, change the <code class="highlighter-rouge">(none)</code> in the ‘Current’ column to <code class="highlighter-rouge">USBCCID</code>. Then click the plus button and pick <code class="highlighter-rouge">USB CCID Payload</code> in the dropdown in the first column. Then you’ll be able to select <code class="highlighter-rouge">ACR 122</code> like above:</p>
<figure class="half centre">
<img alt="Selecting the USB CCID and ACR 122 dissectors in Wireshark" src="/assets/images/nfc-wireshark-dissector2.png" />
</figure>
<p>Wireshark will now be able to work out exactly what commands are being sent to the NFC reader and chipset — you can check the ‘Protocol’ column in the packet list. I’d recommend taking a couple of minutes to click through the packet list and get an idea of what’s going on.</p>
<figure class="half centre">
<a href="/assets/images/nfc-wireshark-packet-list.png" target="_blank"><img alt="An assortment of ACR 122 packets recorded in Wireshark" src="/assets/images/nfc-wireshark-packet-list.png" /></a>
</figure>
<h3 id="fixing-the-bug">Fixing the bug</h3>
<p>After capturing known-good traffic from <code class="highlighter-rouge">nfc-poll</code> and comparing it with the output from Python, I could immediately see where the messages diverged:</p>
<figure class="half ">
<a href="/assets/images/nfc-wireshark-packet1.png" title="The working packet from nfc-poll">
<img src="/assets/images/nfc-wireshark-packet1.png" alt="A packet being examined in Wireshark" />
</a>
<a href="/assets/images/nfc-wireshark-packet2.png" title="The failing packet from my nfcpy port">
<img src="/assets/images/nfc-wireshark-packet2.png" alt="A second packet being examined in Wireshark" />
</a>
<figcaption>The bad packet on the left, with the working packet on the right
</figcaption>
</figure>
<p>The Python packet wasn’t being constructed properly. Some clever hunches with the debugger and breakpoints revealed the source of the error: my attempts at migrating the Python 2 code to Python 3. The original source includes this line:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># brty is type int, initiator_data is type bytearray
</span><span class="n">data</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="nb">chr</span><span class="p">(</span><span class="n">brty</span><span class="p">)</span> <span class="o">+</span> <span class="n">initiator_data</span>
</code></pre></div></div>
<p>Which is no longer valid Python (you can’t concatenate <code class="highlighter-rouge">str</code> and <code class="highlighter-rouge">bytearray</code> objects). I’d migrated it to look like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">data</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="nb">bytes</span><span class="p">(</span><span class="n">brty</span><span class="p">)</span> <span class="o">+</span> <span class="n">initiator_data</span>
</code></pre></div></div>
<p>Once I realised there was a problem here, I decided to actually <strong>read the documentation</strong>, which told me exactly what I should have known all along:</p>
<blockquote>
<p>In addition to the literal forms, bytes objects can be created in a number of other ways:</p>
<ul>
<li>A zero-filled bytes object of a specified length: <code class="highlighter-rouge">bytes(10)</code></li>
<li>From an iterable of integers: <code class="highlighter-rouge">bytes(range(20))</code></li>
</ul>
</blockquote>
<p>So where I was trying to create a <code class="highlighter-rouge">bytes</code> object of length 1 containing the value <code class="highlighter-rouge">brty</code>, I was instead creating a <code class="highlighter-rouge">bytes</code> object <code class="highlighter-rouge">brty</code> elements long with each element set to zero (the same bug is present with <code class="highlighter-rouge">bytes(1)</code>). The easy fix is to make the arguments into iterables: lists or sets. The correct code now looks like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">data</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="n">brty</span><span class="p">])</span> <span class="o">+</span> <span class="n">initiator_data</span>
</code></pre></div></div>
<h2 id="next-steps">Next steps</h2>
<p>Success, right? Not really. The Python 3 port is still incomplete at time of writing and there’s a number of fixes to be made, but progress is definitely underway and it looks like this device is capable of doing what I need it to (but not much more). Once I have the code working for my intended application, I might buy a more capable reader for some further experimentation.</p>
<p>Next week I hope to finish the project for work and provide some code for you that demonstrates how to read and write smartcards.</p>
<div class="footnotes">
<ol>
<li id="fn:image-source">
<p>Image sourced from Advanced Card Systems <a href="#fnref:image-source" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>sigsecI’ve recently been looking into using an NFC reader/writer pad at work to make configuring some devices we have much quicker and easier, as the standard process for these devices is an Android phone and a lot of manual data entry. Checking the device using a mobile is great, but configuring them this way is not scalable.Slack for push notifications2019-03-18T08:45:55+11:002019-03-18T08:45:55+11:00https://blog.sigsec.net/posts/2019/03/slack-for-push-notifications<p>Following on from my post about <a href="/posts/2018/12/push-notifications-with-python.html">Push notifications with Python</a>, I’ve recently been looking at using <a href="https://slack.com/">Slack</a> for more advanced notifications and interactions.</p>
<p>This post will go through the process from the start, including setting up a Slack workspace and making a new App integration. If you’ve already done part or all of this stuff, feel free to skip ahead!
<!--more--></p>
<h2 id="creating-a-new-slack-team">Creating a new Slack team</h2>
<p>Making a new Slack team is free and easy! There are a lot of steps here, but they’re all quick.</p>
<ol>
<li>Go to the <a href="https://slack.com/get-started">getting started</a> page on Slack and click ‘Create a new workspace’.</li>
<li>Enter your email address and click ‘Confirm’. This email address will become the ‘primary owner’ for the team automatically (but this can be changed later).</li>
<li>Slack will email you a code. Type the number into the boxes on the webpage, and it should automatically move you to the next step.</li>
<li>Enter the name for your Slack team and click ‘Next’.</li>
<li>Slack will ask for the name of a project you’re working on — just make one up if you don’t have one. I used “Example project”. Click ‘next’ and Slack will show some confetti and give you a bit of information about what you just did.</li>
<li>After you click ‘See Your Channel in Slack’, you’ll be taken to the Slack web interface where you can start to use Slack.</li>
</ol>
<p>At the top of the page, you should see a nice banner letting you know that you can start to use Slack immediately. There’s also a green button that says “Save Your Account” - click it and follow the next set of steps.</p>
<ol>
<li>You’ll be prompted for your full name and a password to use to secure your account. When you’re ready, click Next.</li>
<li>You can choose to change the workspace name or URL here — <strong>this is the best time to change the URL</strong> if you don’t like the auto-generated one.</li>
<li>Now you can choose to invite other people to your team by email address. There’s also a convenient checkbox below, where you can let people join by themselves if they have an email at your domain. I’d recommend ticking this.</li>
<li>Click ‘Finish’ when you’re done, and Slack will sort out the rest.</li>
</ol>
<h2 id="setting-up-an-integration">Setting up an integration</h2>
<h3 id="creating-a-new-app">Creating a new App</h3>
<p>Now we can start to set up an integration that will let us post messages into our new Slack team:</p>
<ol>
<li>Once you’re logged in, go to the <a href="https://api.slack.com/">Slack API documentation</a> site. In the middle of the banner at the top is a big, green ‘Start Building’ button. Click it.</li>
<li>You’ll now get prompted to create a Slack app. Enter an app name (it can be changed later) and pick the workspace we just made as the “Development Slack Workspace”. Click ‘Create App’ when you’re ready.</li>
</ol>
<p>You’re now at the dashboard for your new app. There’s a lot of choices and options here, don’t be overwhelmed! Before we can do anything cool, we’ll need to add some permission requests and install the app into our workspace.</p>
<h3 id="permissions-and-installation">Permissions and installation</h3>
<p>At the app dashboard, click ‘OAuth & Permissions’ in the sidebar on the left:</p>
<figure class="third centre">
<img alt="The Slack app configuration menu" src="/assets/images/slack-api-app-menu.png" />
</figure>
<p>The ‘Scopes’ heading on this page is where we’ll be adding some Permissions to our app, which allows us to interact with Slack workspaces. I’d suggest adding:</p>
<ul>
<li><code class="highlighter-rouge">chat:write:bot</code>, so you can send messages straight to public channels</li>
<li><code class="highlighter-rouge">bot</code>, as this covers a wide range of use-cases and allows for some more advanced functionality.</li>
</ul>
<figure>
<img alt="The permissions I've granted to my Slack app" src="/assets/images/slack-api-app-permissions.png" />
</figure>
<p>Click ‘Save Changes’ when you’re done. The next link in the sidebar we’ll visit is <em>Bot Users</em>. On this page, click ‘Add a Bot User’. The default values should be sufficient here. Click ‘Add Bot User’ at the bottom to save the configuration.</p>
<p>Lastly, navigate to the <em>Install App</em> item in the navigation menu on the left. Here, click ‘Install App to Workspace’ and then ‘Authorize’. You’ll be given a couple of access tokens; make a copy of these somewhere secure as we’ll need them to connect to Slack and send messages.</p>
<p class="notice--info"><strong>Tip:</strong> Treat these access tokens like passwords. They are the ‘keys to the kingdom’ for your bot and can grant read and write access to pretty much your entire Slack workspace. Don’t commit them to git and be careful where you paste them.</p>
<h3 id="sending-your-first-messages">Sending your first messages</h3>
<p>Here’s some simple Python code using the <a href="https://github.com/os/slacker">Slacker</a> module that will get you up and running with posting messages into a channel (here I’m using <code class="highlighter-rouge">#general</code>). You’ll need the ‘Bot user access token’ given to you in the last step.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">slacker</span> <span class="kn">import</span> <span class="n">Slacker</span>
<span class="n">slack</span> <span class="o">=</span> <span class="n">Slacker</span><span class="p">(</span><span class="s">'xoxb-BOT-TOKEN-HERE'</span><span class="p">)</span>
<span class="n">slack</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">post_message</span><span class="p">(</span><span class="s">'#general'</span><span class="p">,</span>
<span class="s">"Hi sigsec readers! I'm your first Slack integration message :smile: :tada:"</span><span class="p">,</span>
<span class="n">as_user</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">slack</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">post_message</span><span class="p">(</span><span class="s">'#general'</span><span class="p">,</span>
<span class="s">"This time I'm sending with `as_user=True` :robot_face:"</span><span class="p">,</span>
<span class="n">as_user</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</code></pre></div></div>
<p>Here’s how it looks in the Slack app! You can see I’m already including some basic formatting; the <a href="https://api.slack.com/docs/message-formatting">documentation</a> and <a href="https://api.slack.com/docs/messages/builder">message builder</a> are awesome resources.</p>
<figure>
<img alt="Two Slack messages sent by my app using different methods" src="/assets/images/slack-api-app-messages.png" />
</figure>
<p>The difference between the two messages is the value of the <code class="highlighter-rouge">as_user</code> parameter. At this early stage the difference is mostly cosmetic, but there are some differences behind the scenes that will become important later.</p>
<ul>
<li>When <code class="highlighter-rouge">as_user=False</code>, the message will be sent by your <em>app integration</em> (part of the <code class="highlighter-rouge">chat:write:bot</code> scope)</li>
<li>When <code class="highlighter-rouge">as_user=True</code>, the message will be sent by your <em>bot user</em> (part of the <code class="highlighter-rouge">bot</code> scope)</li>
</ul>
<p class="notice--info"><strong>Tip:</strong> Bot users can only post into channels that they have been invited to. If the bot user isn’t in the channel you’re trying to post to, Slacker will fail with an error.</p>
<h2 id="next-steps">Next steps</h2>
<p>From here, the Slack world is your oyster! With these basic steps, you can take your integration wherever you like. You could write success or status notifications for tasks that you’d like to automate. As an example, I added a job to <a href="/posts/2018/12/setting-up-jenkins-in-aws.html">Jenkins</a> that sends me a message with the weather forecast every morning.</p>
<p>An upcoming post will go into detail on how I accomplished this, as well as how you can use more advanced formatting in Slack messages.</p>sigsecFollowing on from my post about Push notifications with Python, I’ve recently been looking at using Slack for more advanced notifications and interactions. This post will go through the process from the start, including setting up a Slack workspace and making a new App integration. If you’ve already done part or all of this stuff, feel free to skip ahead!SMS conversations with Python2019-03-11T07:31:58+11:002019-03-11T07:31:58+11:00https://blog.sigsec.net/posts/2019/03/sms-conversations-with-python<p>Following on from my last post about <a href="/posts/2019/03/sending-smses-from-python.html">sending SMSes with Python</a>, we’re now going to look at ways to get SMS responses and how we can use this to build a basic conversation flow. I find I learn best when I have a goal or project, so here I’ll use the example of an SMS-controlled stopwatch.
<!--more--></p>
<p>If you haven’t read the first article yet, I’d strongly recommend going through it. It’s a reasonably short read and it will cover all the basics that I expand upon here.</p>
<h2 id="response-architecture">Response architecture</h2>
<p>There are two mutually-exclusive<sup id="fnref:mutually-exclusive"><a href="#fn:mutually-exclusive" class="footnote">1</a></sup> methods we can use to get the response SMS data from the Telstra API. We can either:</p>
<ul>
<li>Use polling, by calling the <a href="https://dev.telstra.com/content/messaging-api#operation/Retrieve%20SMS%20Responses">Retrieve SMS Responses</a> API command; or</li>
<li>Use a push method, where Telstra will send a request to a web server we choose whenever a text message is received.</li>
</ul>
<p>Which method we get to use is determined by the <a href="https://dev.telstra.com/content/messaging-api#operation/Create%20Subscription">Create Subscription</a> API method, or more accurately, the value of the <code class="highlighter-rouge">notifyURL</code> parameter we gave it.</p>
<ul>
<li>If <code class="highlighter-rouge">notifyURL</code> is <code class="highlighter-rouge">null</code>, or this is the first time we’re creating a subscription, we use the polling method.</li>
<li>If <code class="highlighter-rouge">notifyURL</code> is a string (our web server address), we use the push method.</li>
<li>If we don’t send the <code class="highlighter-rouge">notifyURL</code> parameter at all, then no change happens and we use whatever we were using before.</li>
</ul>
<p>Don’t worry if you start out with one method and change your mind later. You can adjust the method you use (or change the URL that receives the pushes) by sending a new ‘create subscription’ message to the API with a new value for <code class="highlighter-rouge">notifyURL</code> set as above.</p>
<p>In this article, we’ll use the polling architecture because it requires no additional infrastructure and is easy to get started with.</p>
<h2 id="polling-for-messages">Polling for messages</h2>
<h3 id="the-poll-code">The poll code</h3>
<p>Using the <a href="https://dev.telstra.com/content/messaging-api#operation/Retrieve%20SMS%20Responses">Retrieve SMS Responses</a> API call, we can fetch the <em>next</em> message waiting for us in the queue. If there are multiple messages queued up, we will need to call this method multiple times. The API will indicate when the queue becomes empty.</p>
<p>The code for this is very straight-forward, based on the API documentation and the code from the previous article:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_sms</span><span class="p">(</span><span class="n">token</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">telstra_request</span><span class="p">(</span><span class="s">"messages/sms"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">token</span><span class="o">=</span><span class="n">token</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s">'GET'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">RuntimeError</span><span class="p">(</span><span class="s">"Bad response from Telstra API! "</span> <span class="o">+</span> <span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</code></pre></div></div>
<h3 id="the-api-response">The API response</h3>
<p>The <code class="highlighter-rouge">get_sms</code> function returns a dictionary. If there are no messages waiting, the response is very simple:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="s">"status"</span><span class="p">:</span> <span class="s">"EMPTY"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If there is a message waiting for us, there are more keys in the dictionary. Here’s the dictionary representation of a text message I sent to my subscribed number:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="s">"status"</span><span class="p">:</span> <span class="s">"RECEIVED"</span><span class="p">,</span>
<span class="s">"destinationAddress"</span><span class="p">:</span> <span class="s">"+61472xxx316"</span><span class="p">,</span>
<span class="s">"senderAddress"</span><span class="p">:</span> <span class="s">"+614xxxxxxxx"</span><span class="p">,</span>
<span class="s">"message"</span><span class="p">:</span> <span class="s">"Hi Python! This is an SMS. "</span><span class="p">,</span>
<span class="s">"messageId"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span>
<span class="s">"apiMsgId"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span>
<span class="s">"sentTimestamp"</span><span class="p">:</span> <span class="s">"2019-03-04T15:33:10+10:00"</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="the-handler-loop">The handler loop</h3>
<p class="notice--info"><strong>Take note:</strong> in the Github code for this post, I’ve added a <code class="highlighter-rouge">try ... except</code> block around the <code class="highlighter-rouge">get_sms</code> call, to prevent our entire program from crashing if the API returns an unexpected result. It has been omitted here for readability.</p>
<p>To make sure we deal with <em>all</em> incoming messages, and not just the most recent one, we’ll need to put our <code class="highlighter-rouge">get_sms</code> code into a loop. I use the function <code class="highlighter-rouge">handle_message</code> (which we haven’t defined yet) to take some action on each incoming message.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">handle_all_messages</span><span class="p">(</span><span class="n">token</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">get_sms</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
<span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="s">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s">"EMPTY"</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">handle_message</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="stopwatch-example">Stopwatch example</h2>
<p>At this point, all of the framework code needed to handle receiving messages is complete; the last function to implement is <code class="highlighter-rouge">handle_message</code>. I’ve chosen to implement a stopwatch, as it’s easy to understand and code, and requires us to consider a few different logical conditions.</p>
<h3 id="methodology">Methodology</h3>
<p>We can use the Python <code class="highlighter-rouge">datetime</code> type to track a particular point in time. First, we capture the present moment as our start time using <code class="highlighter-rouge">datetime.now()</code>. Using that stored value, we can see how much time has elapsed by comparing it with the latest value in <code class="highlighter-rouge">datetime.now()</code>.</p>
<p>If we also capture an end time, we can store the total time and refer back to it later.</p>
<h3 id="stopwatch-functions">Stopwatch functions</h3>
<p>We’ll support three basic commands: <code class="highlighter-rouge">START</code>, <code class="highlighter-rouge">STOP</code> and <code class="highlighter-rouge">TIME</code>. I’ve written the demo code to reply to the user after each message with a brief summary of the action we’re taking, so we know the server is still listening and as practice with our SMS sending code.</p>
<h4 id="start">START</h4>
<p>If the timer is stopped, this will start the timer and notify the user. If the timer is already running, this will reset the elapsed time to zero and tell the user that the timer has restarted.</p>
<h4 id="stop">STOP</h4>
<p>If the timer has never been started, we’ll tell the user to start it first. If the timer is currently running, this will stop it and notify the user with the elapsed time. If the timer is already stopped, it will notify the user that it is already stopped.</p>
<h4 id="time">TIME</h4>
<p>If the timer isn’t running, we’ll tell the user to start the timer first. If the timer has ended, we’ll tell the user the final measured time. If it’s still running, we’ll tell the user how much time has passed since we started.</p>
<h3 id="the-logic-code">The logic code</h3>
<p>The complete Python code for this function is quite long, so I’ve simplified it for display purposes here in the blog by removing the code that sends replies to the user. The code in Github includes these replies.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">STOPWATCH_START</span> <span class="o">=</span> <span class="bp">None</span>
<span class="n">STOPWATCH_END</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">handle_message</span><span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="k">global</span> <span class="n">STOPWATCH_START</span><span class="p">,</span> <span class="n">STOPWATCH_END</span>
<span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="s">'message'</span><span class="p">]</span> <span class="o">==</span> <span class="s">"START"</span><span class="p">:</span>
<span class="n">STOPWATCH_START</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">STOPWATCH_END</span> <span class="o">=</span> <span class="bp">None</span>
<span class="c1"># tell user that timer is restarted
</span>
<span class="k">elif</span> <span class="n">message</span><span class="p">[</span><span class="s">'message'</span><span class="p">]</span> <span class="o">==</span> <span class="s">"STOP"</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">STOPWATCH_START</span><span class="p">:</span> <span class="c1"># if we haven't started yet
</span> <span class="c1"># tell user to start timer first
</span> <span class="k">elif</span> <span class="n">STOPWATCH_END</span><span class="p">:</span> <span class="c1"># if we've already ended
</span> <span class="c1"># tell user we are already stopped
</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># we are currently running
</span> <span class="n">STOPWATCH_END</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="c1"># tell user how long timer was running for
</span> <span class="c1"># based on (STOPWATCH_END-STOPWATCH_START)
</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="s">'message'</span><span class="p">]</span> <span class="o">==</span> <span class="s">"TIME"</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">STOPWATCH_START</span><span class="p">:</span> <span class="c1"># if we haven't started yet
</span> <span class="c1"># tell user to start timer first
</span> <span class="k">elif</span> <span class="n">STOPWATCH_END</span><span class="p">:</span> <span class="c1"># if we've already ended
</span> <span class="c1"># tell user how long timer was running for
</span> <span class="c1"># based on (STOPWATCH_END-STOPWATCH_START)
</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># we are currently running
</span> <span class="c1"># tell user what timer is currently at
</span> <span class="c1"># based on (datetime.now()-STOPWATCH_START)
</span>
<span class="k">else</span><span class="p">:</span> <span class="c1"># unknown command
</span> <span class="c1"># tell user we only know 3 commands
</span></code></pre></div></div>
<h3 id="the-script-main-code">The script ‘main’ code</h3>
<p>To get all of this to actually <em>run</em> when we execute our code in Python, we need to put it into a loop. I don’t want to burn up the CPU on my laptop, so I’ve made it wait a second between each check for replies. This means our stopwatch is pretty inaccurate, but it’s just an example.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">sleep</span>
<span class="n">token</span> <span class="o">=</span> <span class="n">auth</span><span class="p">(</span><span class="s">"PutYour32CharClientKeyHereThanks"</span><span class="p">,</span> <span class="s">"ClientSecretHere"</span><span class="p">)</span>
<span class="n">create_subscription</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
<span class="n">send_sms</span><span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="s">"+61400000000"</span><span class="p">,</span> <span class="s">'Stopwatch app online! I understand "START", "STOP" and "TIME".'</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">handle_all_messages</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
<span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="seeing-it-in-action">Seeing it in action</h3>
<p>Here’s how it looks on my phone:</p>
<figure class="half centre">
<img alt="The SMS conversation on my phone" src="/assets/images/python-sms-conversation.jpg" />
</figure>
<h2 id="closing">Closing</h2>
<p>As before, I hope you’ve been able to learn something interesting and useful from this post! The complete Python code shown here can be found <a href="https://github.com/sigsec/blog-examples/blob/master/telstra_messaging_api_conversations.py">in Github</a>.</p>
<p>I’ll take a moment to note that this code has a lot of limitations.</p>
<ul>
<li>The stopwatch state is tracked at a global level, so if we tried to use the same architecture for multiple recipients we’d end up with everyone trying to fiddle with the same stopwatch. A better architecture is to track the sender of the message (<code class="highlighter-rouge">message['senderAddress']</code> in our <code class="highlighter-rouge">handle_message</code> function) and provide a per-user stopwatch. This could be a great exercise for you to test your understanding.</li>
<li>The stopwatch is not very accurate. The delay in the main loop is necessary to stop our Python program stealing all of the computer’s resources, but it also means we’re probably going to be inaccurate in our stopwatch times by up to 5 seconds once network delays are included.</li>
</ul>
<p>Lastly, I was originally hoping to include a section here on the push method of handling replies, but just covering polling for replies is already a lot of code. I’m interested in covering the push method in the future, but it might need a couple of other foundational posts first before I get to it.</p>
<div class="footnotes">
<ol>
<li id="fn:mutually-exclusive">
<p>Either one <em>or</em> the other, but not both <a href="#fnref:mutually-exclusive" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>sigsecFollowing on from my last post about sending SMSes with Python, we’re now going to look at ways to get SMS responses and how we can use this to build a basic conversation flow. I find I learn best when I have a goal or project, so here I’ll use the example of an SMS-controlled stopwatch.