Le Roux Bodenstein

I’ve been working with Sharp’s LS013B7DH03 memory LCD. While looking for a driver to use with MicroPython I realised that they all invert some COM signal every time they update the display, usually with a vague comment about how you have to update the display regularly. The EXTCOMIN and EXTMODE pins also baffled me until I eventually found this handy bit of documentation by Peter Hinch that explains it.

So basically one way or another the COM signal has to be inverted regularly or you’ll gradually damage the LCD. I agree that doing that in software seems to be easier. For my application I want to keep the ESP32 in deep sleep as long as possible and waking from deep sleep takes a significant part of a second, so waking every second or so to invert the signal is out of the question.

The ESP32 also has a ULP (Ultra Low Power) Coprocessor, so I got the idea to use that to periodically invert the COM signal. But then I thought.. Is it possible to just have the entire display driver live in the ULP processor with the entire display buffer in the 8K of RTC memory along with the code? And the answer turns out to be.. kinda. But it is just a bit too slow.

The ULP is extremely limited. It has 4 general purpose registers, a weird hidden 8-bit register that’s only really useful for loops and very few instructions. It is clocked from the ±8Mhz clock. Seems like you can maybe also clock it from 40Mhz/4 = ±10Mhz but I haven’t figured out how to do that yet. Instructions take 4, 6, 8 or 12 clock cycles to execute and load the next instruction. The macros for reading/writing peripheral registers take who knows how many more. It can only access memory at 32-bit word boundaries, but it can only read/write to the lower 16 bits. The upper 16-bits of each 32-bit word is inaccessible.

The display is 128x128 monochrome, so a full screen buffer is 2048 bytes, but because half the memory is inaccessible it actually takes 4096 bytes, or 1024 32-bit words. ie. exactly half of the 8K of RTC memory. The ULP is designed to do ADC readings or monitor something over an I2C bus or maybe check some pins and then wake up the main processor. The display talks SPI, the ULP has no access to SPI hardware. So I would have to bitbang it..

I’ve never written anything in assembly before (let alone for such an obscure target), so I had no feel for what kind of speed I’d get. I decided to just wing it and if it doesn’t work out I could always go back to plan A which was to only run the ULP while the main processor is asleep and have the main processor take over the SPI bus to quickly update the display otherwise.

So my driver borrows from MicroPython nano-gui’s driver in that it subclasses framebuf.FrameBuffer, but all the Python code does is it has a 2048 byte screen buffer that it copies over to the RTC memory in the “only use half of it” weird style that the ULP understands. This way it can behave just like a normal MicroPython framebuffer display driver. The ULP code basically then just stores the last COM signal, wakes up periodically, inverts it and copies the buffer contents to the display from that patch of RTC memory. Then goes back to sleep.

I optimised this a fair amount and.. it takes about 140ms-ish according to my oscilloscope to copy over a frame. I can maybe get that down closer to 100ms which would be 10 frames per second and I could maybe boost that by a further 20% if I can switch to a faster clock, but that’s still gonna be in the range of 10 FPS for full screen updates which isn’t great. It should be possible to speed up “off” pixels by a bit and to make it only update changed lines, but I’m not crazy about those limitations - I like things to perform predictably.

This was interesting and I learned a lot. I put my hacky code up as a Gist.

What I’ll rather end up doing would be to copy the frame data to the display using the main processor from the main program and then only use the ULP during deep sleep with the “maintain memory internal data” mode where you send it one byte including the inverted COM bit. I was hoping to avoid that because that has different bits of complexity: You have to make sure that control of the SPI line gets properly handed off between the main processor and the ULP.

I recently put up a greenhouse in our backyard. So naturally I’d like to start logging temperature, humidity and maybe soil moisture or solar radiation levels, maybe take some pictures for creating a timelapse of the plants growing through the seasons, perhaps even automate watering the plants or opening and closing the window..

So I started looking into options for doing this. As I’ve found several times before every electronics project seems to start with (and largely revolve around) how to power it.

The way I see it there are several options here:

1) Lay a power cable from the house to the greenhouse. I’m renting the property so don’t know about making permanent changes. Also not clear about whether that would require planning permission.

2) Power it with a battery and then replace the battery when it runs out. With some basic power saving techniques even a small battery would easily last a month. But this just feels inelegant and not like a lot of fun.

3) Keep a battery charged with solar power.

4) Keep a battery charged with a small wind turbine. This is a fairly small backyard and I don’t know how my neighbours would feel about a wind turbine.

5) Skip the battery and charge a supercapacitor with a small solar panel. This is actually the cheapest option, in some ways the trickiest but also the one I find most interesting.

Solar panels have become ridiculously cheap and seem to still be on a technology curve, so still no end to the price drops in sight. Small (less than 10 watt) panels are now only a few dollars each.

Small supercapacitors are also surprisingly affordable. A downside of supercapacitors is the low maximum voltage of typically around 2.7V for EDLCs. This is a bit low for powering 3.3V electronics. You can put them in series to increase the voltage, but it is necessary to balance them to stabilize the voltage due to slight differences in capacitance and ESR caused by manufacturing tolerances. This would slightly increase the complexity of working with them. Alternatively you could use a boost converter that can start up at a very low voltage, but this increases the cost.

However, you can now buy 5.5V or 6V supercapacitors that actually have multiple inside them in series. I’m not sure if they contain balancing circuitry or if they are just “matched pairs”, but the point is that that becomes an implementation detail that’s already taken care of for you. The higher voltage is nice because then you can charge them with USB, power 5V electronics directly (AVR microcontrollers can handle up to 5.5V) and much more of their charge becomes useful or accessible without needing a boost converter.

You also get ultra low dropout (say 350mV or even less) linear regulators that are much cheaper and simpler than a buck/boost converter.

One inefficiency is that you ideally want MPPT charging to get the most out of the solar panel, but you can also just overspec the solar panel. In this kind of application it is bound to be cheaper and even a couple of watts is already overkill for what I’m doing here anyway.

For a low power WiFi and camera module the obvious choice is an ESP32-CAM which (somehow) costs less than $10 each. It runs on 3.3V. It has a 5V input with and “LDO”, but “low” here means +/-1.2V, so not very useful for our purposes so we’ll have to bypass it and power it with 3.3V directly. I’d like to measure the supercapacitor’s voltage which requires ADC, but all the esp32’s ADC pins are already taken up. There are hackish ways to use some of the remaining pins or to repurpose the LED’s pin, but I’m not going to go there. Besides - according to the “datasheet” for the ESP32-CAM it draws 6mA of current even in deep sleep. This is unreasonably high. I didn’t bother testing it myself because I figure they are unlikely to advertise figures that make it sound worse than it really is.

I came up with this solution:

1) A 5.5V maximum (once you include a blocking diode’s drop) solar panel charges a 5.5V (or higher) supercapacitor.

2) The supercapacitor directly powers a 5.5V capable microcontroller. I like the new “mega” attiny series.

3) The attiny sleeps just about all the time, but wakes up every minute or so (configurable) and checks its supply voltage.

4) If the supply voltage is high enough (meaning the supercapacitor is reasonably charged) it turns on power to an LDO that powers the esp32-cam through a P-Channel MOSFET high side switch.

5) The attiny immediately goes back to sleep.

6) The ESP32-CAM reads the temperature and humidity from a sensor and takes a picture. It then asks the attiny for the supply voltage over i2c.

7) The attiny wakes up from the i2c connection, reads the voltage, sends it and goes straight back to sleep.

8) Then the ESP32 sends the temperature, humidity, voltage and photo to a server over wifi.

9) Once it is done it asks the attiny to turn it off, again over i2c.

10) The attiny wakes up from the i2c connection, toggles the power to the esp32-cam and goes back to sleep.

I breadboarded this and it works remarkably well. The entire cycle uses very little power. Everything is as “off” as it can be as much as possible. I still need to measure it, but over time this averages out to well under 1mA even though the esp32 uses hundreds of milliamps during the moment it is fully powered and transmitting over wifi.

An obvious optimisation would be if the server could tell the esp32 about sunset and dawn times and it could then communicate back to the attiny how long it should let the esp32 cam sleep before it powers it down. Because at night it would be too dark for timelapse pictures anyway and it would just drain the supercapacitor before the morning. Alternatively we could still log all the stats through the night and just not take any pictures.

I plan to design a PCB and enclosure to turn this into a proper project. Will build it and report back ;)

As technically minded people we often get ideas for projects, products or similar and it can be easy to get carried away. I often design things in my head, think about how I’d build it, what it would involve, what it would cost to make, how I’d get it made, what programming it might involve, how I’d do that, etc. This is just something I enjoy doing and for whatever reason it is the kind of thing my mind often does when I should be sleeping or doing other things.

Sometimes my ideas progress further and become projects and I can easily get lost in them. Then eventually I start to think about whether this would be useful to others or is there even a market for it and the doubt starts to creep in. And often that’s where I end up losing the drive to complete the project.

There are of course many reasons for taking on a project. If you just want it for youself, then that can be a fine goal. If you’re doing the project so you can learn some new skill in the process then that’s excellent too.

But I’m now old enough to realise that I have to pick my battles. And there are enough project ideas out there where I can potentially have it all: Learn something new, get something useful or fun I can use AND maybe end up with something useful to others or maybe (just maybe) even a viable product.

So I’ve added this to my checklist for before I start on yet another hobby project: “What if I succeed?”

For product features I think teams should ask the same thing: “What if we succeed?” It is so easy to get lost in discussions about whether something is possible, how much work it would be, how long it would take to implement, etc. that people often don’t pause to think whether they should be adding this feature at all. Often whatever you do you do at the cost of everything else you could be doing. And features are never free - they often have to be maintained and always add complexity.

Something I’ve been thinking about on and off for a while now is how important it is for each project that the people in charge have good taste. I realise that this kind of discussion is a minefield because it immediately comes across as an accusation, but hear me out.

What is taste? Taste is unfortunately subjective. It is a bit “I know it when I see it”. It is not the same as design, but there is certainly overlap. Although thinking about it in fact I think you can argue that it is design in the broader, purer sense.

Taste is often about saying no. It is about pushing back. It is about simplifying. Sometimes it involves asking rather pointed questions. It can be making decisions. Paring back. Thinking about the bigger picture and the overall direction. Rethinking past decisions and reconsidering alternatives when things change.

It is an important tool to guard against feature creep and bloat or to just stop things getting out of hand and becoming a mess. It is essential for staying focused.

Taste is what separates great leaders, products, teams or companies from the rest.

Once about 10 to 15 years ago I was visiting my parents. I was sitting with my laptop in their lounge, typing code into a text editor. I can’t remember the details, but it was either for a pet project or I was fixing some code to do with my startup at the time. And my father walks in and asks “How do you know what to type?” I gave him some off the cuff answer like “I don’t know. When you talk, how do you know what to say?”. He wasn’t happy with the answer and walked off while I continued coding or debugging whatever it was I was working on.

I’ve been thinking about this at least once a month ever since.

To me this question comes back to bigger questions like how our brains even work. How do we learn? How do we think? When you learn to speak and later to read and write or to do maths, what really happens in your brain? How do our brains reason about things? How do we solve puzzles and other problems? How do we plan things?

All I know is that we probably don’t know, but whatever it is, it is the core skill of many technical careers.

I’ve described myself at times as a programmer, a developer or a software engineer. I think it is all different ways to say the same thing. And I think what it is we do is to figure things out, break it down, understand all the parts of it and approach how the whole thing works or how to split it all up into parts that can be tackled bit by bit and then eventually arrive at a complete system that hopefully solves the problem you set out to solve.

But if you think about it, this is also what all different kinds of engineers and related technical professions do. Consequently you’ll find that people in these professions tend to have lots of other technical hobbies and side projects. And that the best people in these professions could relatively easily reskill and move to one of the others. I’d go as far as to say that they could (given enough time and motivation) learn to do anything.

Or at least I like to think that I could learn to do anything. Or that I could have made a success in all sorts of careers if I made different choices or had different preferences when I left high school.

Does everyone have this innate ability? Is it genetic? Is it something you gained due to your upbringing? How does this “core skill” of not just learning but knowing how to learn (for lack of a better term) really work? And can it be taught? I have thought about this a lot and I still have no idea.

Clearly humans are born with some innate ability to learn. Whatever that means. I don’t think humans usually learn by rote learning. I think something much more low level happens in our brains. Connections at a deeply subconscious level are made that allows our brains to just arrive at the answers. Our conscious brains then like to post-rationalise how we got there. That’s my theory, anyway.

Years ago I read somewhere “You have to be interested in order to be interesting.” which can be interpreted at least two ways, but I’ve always taken it to mean that you have to be interested in the world and have interests so you would read up about and learn about those or otherwise chase and practise those interests and then you would be interesting because by the end you become an expert (or at least versed) in those things and then maybe once you can hold a conversation about it you’re more interesting as a result. Maybe this is how people learn?

Or maybe it is all about experience? I’ve seen it argued that people learn by doing. So once you’ve solved (or failed to solve after trying) many similar problems over the years, maybe you learn how to solve more of the same kinds of things more easily?

But none of this is helpful in answering “How do you know what to type?”

Every now and then I discover test code that cannot fail. There are two main categories:

1) If you comment out the code that performs the action, the asserts still pass.

These tests usually check that something won’t happen. Like let’s say a user disables notifications and then you’re trying to make sure that no notifications go out. The obvious problem is that if you don’t generate notifications, then nothing will go out. If you do generate notifications but they are generated asynchronously and you check too soon, then it will also look like no notifications went out. If your test looked for a specific notification but got the search/filter wrong, then it will also look like no notifications went out. Regardless of whether they did or not. So testing this requires careful thinking and to some degree you need to prove that your test even works. Like maybe before disabling notifications you enable it, generate a notification and look for that notification so that the test that tests that disabling notifications can do the exact same thing when it proves that disabling notifications worked. Then you can have some confidence that the test is valid.

Experience can help anticipate these situations. Peer review hopefully helps if you have good peer reviewers. It helps being extremely skeptical when working on or reviewing this sort of thing.

2) Everything is stubbed to the point where you’re only testing the stubs.

These tests are usually easy to spot on those days where 300 of 320 tests failed because something fundamental broke and you start wondering why the other 20 passed ;)

There is probably a whole separate post about when to stub or mock and when not to, but the TLDR is to only stub as a last resort. Prefer integration tests over unit tests and prefer real data over fake data for tests. If it feels hard or like too much work or something that’s really slowing down test writing or even test execution, then take a step back and work on the infrastructure or the test helpers, maybe try and think outside the box.

In the meantime check out these classic talks on the subject if you’re interested:

• www.youtube.com/watch
• blog.testdouble.com/talks/201…
• blog.testdouble.com/talks/201…

We use hapi for the project I work on. The decision was made before I joined and at first I was mildly grumpy about it. My reasons were silly like how I’d rather be gaining experience in the most famous/popular libraries and frameworks and at first I disagreed with some of the style decisions. The way only programmers can get really upset about the tiniest things.

Maybe it is just stockholm syndrome, but I really started liking it and also appreciating it more and more over the years. There is just something to be said for a coherent, well thought out vision and for frameworks and libraries that work really well together.

My favourite part of the hapi ecosystem is probably lab. The test coverage calculator is second to none. Linting is integrated and just works. The test runner’s output is really usable and code (the assertion library)’s error formatting is incredibly helpful especially if you’re smart about your test assertions. I use it in my pet projects wherever it makes sense too.

After that joi is unparalleled as far as schema validation libraries go. To say that we use it extensively is an understatement.

You can read the styleguide if you’re interested, but I just want to highlight a few of the things I came around to over the years:

Imported modules names are uppercased and the only other module-level declaration is an object called internals. All constants and non-exported functions go there. This means you can tell exactly where any identifier comes from at a glance. It is either an imported module, an export, internals or something that was declared in the chunk of code you’re working on. I now use this style everywhere and I think it is brilliant.

There are a number of other style rules that make it easy to scan through code like this or that makes code easily “greppable”. Like single quote strings only. Top-level functions are declared via assignment with the function keyword only. Nested functions are declared as arrow functions. No chained declarations. const and let only. camelCase, no underscores. Plural for arrays, single for each element in an array. All scopes get wrapped in {}. Array function arguments must be wrapped in parenthesis. Probably a number of others I can’t think of off the top of my head. The rest are just style choices where you have to pick a way and the exact way you choose matters less than the fact that you made a decision.

I’m a complete convert to 100% test coverage and I now do that everywhere I can.

There are probably more, but those are the main ones that stand out. Really I think it boils down to the overall quality which is just second to none.

Have you ever been on a project for years and it ends up feeling like the same things come up for discussion again and again, and every time everyone argues until the time is up, no decision is ultimately made and no action gets taken so it just inevitably comes up again?

This is usually not one of the big business-facing decisions. There are exceptions, but usually it tends to be a technical argument. Some technology the team wants to adopt or switch to or some changes they want to make to the code. The kind of thing where everyone agrees that something should be done, but no one can agree on the exact details. Management is even happy to allow it in principle. If only the team can agree.

You might consider this controversial, but I don’t think management by consensus and having committees take all decisions will always work. How often does it work? 80% of the time? 90% of the time? 99% of the time? You can argue over the exact split, but eventually there are things that start to pile up over the years. If you can’t make a decision 20% of the time, then you probably have bigger problems. But let’s say it is only 1% of issues that never get resolved. Over the years that 1% will pile up and it becomes all that remains.

Politics is a good analogy here. The laws that the vast majority of people agree on don’t get argued over. But after a few hundred years you end up with the few issues where there’s a near 50/50 split piling up and dividing a nation to the point where it feels like no one can agree on anything.

So for that remaining minority of decisions I don’t think the solution is to persue more discussion, more investigations, etc. Someone in the team has to have the power and the guts to take the initiative and just make a decision. Pick one option and go with it.

Continuing to do nothing is often worse than an imperfect solution.

On the project I’m on we have 20% of our time allocated towards doing “chores”. What are chores? That’s development work that’s not either feature development or a bug fix.

This could mean things like:

• modernising code to work with newer language features
• updating code to work with a newer version of a library
• expanding logging
• expanding our end to end tests
• working on our development tooling
• refactoring code
• … and probably lots of others

This is important to help reduce tech debt. Without this every codebase will get old, outdated, riddled with security issues, builds will break, etc if you don’t constantly keep things up to date, modernise it, fix what’s broken, work on developers’ pain points and so on.

Most projects have unnoficial time for doing these things, but I think every project should have this formalised and developers’ team leads should get full buy-in from management and all corners of the business. If not then it will inevitably lead to problems down the road.

Effective daily standups are crucial to software development especially for remote teams. And in this new world with many more people working remotely there are suddenly many more people forced into daily standup video calls.

I’m lucky to have worked remotely by choice for the last few years and it has taught me one thing about standups: There is no one correct way to do them!

The project I’m on is fairly big. Way too big to cram everyone into one call. So we’re split into different teams. And the teams have self-organised with subtly different culture around things like programming style, peer review, and daily standups. The teams have very different sizes even which is already enough reason structure things differently. Sometimes people move between teams or spend time straddling two different ones and ideas spread between them.

One of them is that the last person to join the call starts with their update and once you’re done with your update you nominate the next person to give theirs. This reduces the awkwardness where no one’s sure when to start or when the previous person is done and who should go next. It also means you don’t need anyone leading the call so even if there’s a clear team lead things can still continue as normal if that person cannot make it. It cuts out a lot of dead time between people talking and helps to keep things moving along quickly.

As a general rule of thumb standups should be quick. Typically you would just mention what you did yesterday, what you’re going to do today and whether you have any blockers. Any tangents or discussions get taken “offline” meaning on slack or in a separate zoom call afterwards so you don’t end up holding up the entire team on a topic only two or three people are interested in.

All of this might make it sound that standups should be as quick as possible and usually this is the case. But for remote teams this might be the only time in a day that people speak to each other. People could be quite isolated otherwise. So there’s a balance to strike between chit-chat before you start and joking around vs being all business all the time. In my experience no team ever fully agrees on where that balance lies - standup length is a frequent retrospective meeting topic.

One thing I’m pretty sure of is that standups shouldn’t be your only meetings. We have regularly scheduled grooming, planning and demo meetings where we can go into more depth. And you can always jump on a quick call if you need to.

How many meetings are too many? That’s a whole other topic..

Modern best practises dictate that you should have automated testing with linting, full test coverage with real data and continuous integration. Development should have been done through peer reviewed pull requests.

But what if a system or some part of the system already exists without all of this in place? Even today it can still be hard enough to convince people of the benefits of following best practises during development. Now imagine trying to convince them of slowing or pausing work on new features so you can get the house in order. Never an easy conversation.

Even if your team has a “chore budget” (and you should!), turning the ship around to focus on code quality might require a massive amount of effort, especially for large established projects.

There is always the temptation to throw it all out and rewrite. True refactors can also be really hard. Can you get there incrementally? Whichever approach the team decides on it will require full buy-in at the highest levels.

But code quality and a codebase you can trust is always worth it in the end.

I’ve been thinking a lot about writing tests for web apps lately which gave me an idea for a series of posts.

First up: Enforcing 100% test coverage.

On the project I’m on we’ve been enforcing 100% test coverage for much of the system for the past few years.

What does this mean?

There are probably multiple ways for test frameworks to calculate test coverage, but the one we use (@hapi/lab) considers all lines in all files in your repo imported by your tests. All the files that are not in your tests. Typically everything in lib/ or src/, but not test/. One subtle thing to keep in mind is it only considers files that are used either directly or indirectly by your tests.

This has the effect that all the code that you test has to meet your specified threshold. You can set thresholds other than 100%, but for this discussion I’m going to focus on 100%.

So in our case, all the code has to be executed by our test suite at least once.

One more subtle thing to consider: Every condition has to be met AND not met. So if you have an if statement like if (a) {, then you need a test for the case where a is truthy and also one for when it is falsy. If you have an if statement like if (a && b), then you need test cases for when 1) a is falsey, 2) a is truthy and b is truthy and 3) a is truthy and b is falsy. ie. Every condition has to be tested. Not all coverage calculators are equally good, but this is the ideal.

What I thought it was about

At first, like many, I grumbled about this. It felt overly strict. You would work on a feature for what felt like ages and got all the tests to pass, but your PR build would still fail because you’re missing coverage on a couple of lines. I’d find myself arguing that this line doesn’t matter. It will never happen in production. We can just add a linter comment to ignore this one. etc.

And almost every time I added tests to fill out that last bit of test coverage I would discover a bug. But that’s not even the most valuable thing. I gradually came to realise that test coverage isn’t about testing at all. Let me explain.

100% test coverage is about code quality

Often when I find that a line misses coverage it ends up being a really complicated statement with lots of ands and ors. And the coverage tool just points at the line, so I don’t know which condition it is that misses coverage. So I do the sensible thing and split that line up into multiple if statements so I can at least tell which part is missing coverage.

A common case would be when checking multiple things to determine one boolean. So the easiest is usually to move it all out to a function. Then I’m forced to think of a name for this function which is already a good thing because good names are part of the way to self-documenting code. Since this function is just going to return true/false I can now use early returns. So I can divide and conquer. if (!something) { return false; }. if (somethingElse) { return false; }. And so on, ending finally with a return true;. (or obviously you can approach it the other way around - whatever makes most sense in the moment).

Now I can run the test suite again and the lines with the missing coverage will point to exactly the condition that was never tested. It is usually obvious whether it is the truthy or falsy path depending on whether the code inside the if statement blocks executed or not.

Already we have ended up with better, much more readable code.

If it is a legitimate case where you’re just missing a test, then write the test. But what if you find yourself arguing that “this can never happen!” or what if you try to write tests for it and realise that you cannot ever get that condition to be either true or false? This often means that the code is redundant and should be removed. It could be that you’re testing things you control, it could be that the input was already validated in an earlier scope, it could legitimately be something that could never happen or maybe another check in the same function already covers it. (All good topics for future posts.)

This, to me, is the true benefit of 100% test coverage. Without it there is a good chance that you would never have stopped to think about any of the above. And even if you did it is unlikely that you would have had as much confidence that you could just remove the code.