21 MAY, 2021

Technology, connections, and IoT / 'smart' home devices

This post is my thoughts in response to an invigorating post I saw. Fair warning, it might sound a bit like a rant too. I'd strongly recommend reading through the post (linked below) before reading on:
Why We Need to Rethink the Computer 'Desktop' as a Concept.

An analogy

This might be moved down a bit later, but before I forget: technology should just work. Together. For an example of where we've managed to kinda do this, is HTTP.

If you're of a certain age and remember a certain time, you've probably typed in http://www. more times than you can count. But that simple sequence of characters powers the entire World Wide Web! It's a modern marvel, as I'm sure you've been told many times, so let's move on to my point. It works. All of it, together*. (*in theory)

For instance, I can use any browser on any device to access any website run on any server, and it will work. Why can't we do that with everything else that's tech? Screw "the ecosystem," why do they have to be exclusive? Why can't everything be part of "the ecosystem"? Let's talk about it...

The bigger problem

If you're a bit of a techie person, I'm sure you're aware of this phenomena: normal people are hopelessly oblivious to tech. I know several people my age, who genuinely don't know how to use a mouse and keyboard. I have to tell them about what a right-click is and does! They've only ever used a phone, maybe a laptop occasionally, and thus lack basic computer intuition. And it goes beyond the hardware, to the software even more so...

I often hear people complain of software doing this or not doing that, to which I often say, "Why don't you just ..." It's usually a very simple thing, like them getting popups from some outdated or unwanted software bloatware. My answer is quite simple, always the same: turn it off. Or, "just uninstall it." To which there are two responses: "Wait, you can do that? I don't know how." and "I can't be bothered."

And I think to myself, 1) how do you not know that you can do that, and 2) but it's so easy! (respectively) But from their perspective, they've got a point, based on experience. Managing their system is a painful experience for them. For us, it's no problem to run up the uninstaller, click through once or twice, wait, and then "Finish" it off. No more annoyances. For them, it's a whole other thing...

They have to find the uninstaller, probably under an obscure name in the a specific list in the Control Panel (and they probably don't even know what the software is called to begin with), then realize you need to select it and click one of the options on the top bar (but not the very top bar), follow a confusing popup window ("I already said I want to uninstall it!"), and then, they even have to reboot their computer.

No wonder it's a horrific ordeal for them. And Microsoft hasn't done a lot to make it better either. Compared to their phone (or even macOS/Linux), all they need to do is drag the app icon to a bin, or hold and tap the 'remove' button. Maybe a confirmation, and it's gone. No reboots, no further uninstallation wizard, no auto-website survey popups- it's just gone.

So what's a good example of this?

As usual, when we look to an idealised world, we end up with sci-fi. In this case, I'd like to take note of two examples: one famous, and one that a lot of people obviously want.

Asgard

"Whosoever holds this hammer, if he be worthy, shall possess the power of Thor." - Odin

I'm sure we all remember this classic line from Thor. Odin seeks to humble his wayward, headstrong son by banishing him, after stripping him of his power. So, what's special here? Well, to see that, we need another quote...

"Your ancestors called it magic, but you call it science. I come from a land where they are one and the same." - Thor

When Odin is whispering to Mjolnir, sure it's a bit of magic, but as Thor insinuates, it's also technology. Shall we analyze what's happening here, then?

Wow. That's a lot of cool stuff for a single shot in a movie. And that's just stuff that's implied to happen, which we only learn about later in the film! Now, if only "smart" speakers were actually that smart and capable... This is the world I kinda want to live in (and why can't I?)

Iron Man

Throughout the Iron Man and MCEU films, we've seen a lot of Tony Stark's work. And a lot of his cool tech. But I want to focus more precisely on Iron Man itself. As in, the Iron Man concept/suit. It's incredible tech to begin with, being a very capable (to the point of physical impossibility) exoskeleton, but I'm more interested in the Human Computer Interaction (HCI) again.

The Iron Man suit is operated by, and the HCI makes heavy use of, an AI. For the first several occurences onscreen, the suit is run by JARVIS. JARVIS's primary interface appears to be via voice, although he is also monitoring the user via all the available sensors and devices.

There's a particularly good moment in Iron Man 2 where Rhodes (a.k.a. War Machine) steals the Iron Man suit. Of note is the comparison between this and Tony Stark's first flight attempts in Iron Man the first. We get a montage of failed flying attempts, before he manages to simply hover successfully. This implies some level of skill, and JARVIS's intervention via (presumably) fly-by-wire controls.

But when Rhodey gets in the suit, he is immediately able to operate it, and even (shakily) does a short, controlled flight/hover! He finishes the scene by using the fire control systems to target and shoot Tony Stark, in another suit. All this is done within minutes of donning the the second-generation exoskeleton. That's an impressive learning curve, especially given it's' done in a stressful situation (hand-to-hand combat with bystanders).

We see similar intuitiveness with Tony Stark's other tech (holograms for instance) that he has had a hand in creating. He proves himself a master of intuitive, connectedness. His whole house is genuinely smart being fully controllable and controlled by an advanced AI, which has access to everything within it.

It's capable of this by very precise tracking and recognition of Tony, and he interacts seamlessly and flawlessly with the holograms (even to the point of the trash bin being a basketball hoop across the room). This level of tracking and precision just isn't generally available, much to the detriment of modern technology... (smart homes could get a whole lot smarter then)

So, how should it really work?

Screw accounts, passwords, pins, 2FA, platforms, "ecosystems" and all that junk. That's a holdover from a more primitive time. We really should be drastically moving on from it. You're you. That should be enough to use any and all technology that you can.

And more than that, each device should communicate with each other device. No no, not each device that is fully updated and made by the same manufacturer within the 2 year support period. Every device should work together.

So let's talk practical now: I've got a "smart" home device or two. I've got experience setting "smart" home stuff up. I'm also a very technical person. If I have a Raspberry Pi that I want to control all cool like, I'll make a UWP app that integrates with Cortana, so I can yell "Hey Cortana, turn Raspberry Pi on." and it works (not anymore, grrr).

To set the scene, I have a Raspberry Pi; I have a desk; I have a USB LED strip running around my desk.

Stage one: LEDs plugged into Pi

It started simple. I had the strip plugged into a USB port on the Pi. The Pi ran a web server, and I could click buttons, which would send some POST requests to PHP files, and execute a shell script to turn the USB port on or off. I used Kustom to build an Android widget for my phone, so I could easily send POST requests and control it that way as well. Lastly, I made a UWP app as mentioned above, so I could talk to Cortana. That subsequently doesn't work anymore, thanks Microsoft for gelding your own product (again).

Stage two: Google Home/Assistant/Nest mini

I received a Google Nest Mini speaker as a gift. It's a neat little device, if a little bit lackluster in feature and usefullness (as all "smart" home tech is). After a relatively smooth setup via the Google Home app, I was off to the- oh. My brand new "smart" speaker doesn't actually work. Well it does, but it's not smart. I can't control my lights with it.

HTTP POST requests are the staple of interconnected working-ness, in terms of making stuff actually work across a massively wide range of devices and software and environments. It's an industry staple. And my "smart" speaker don't do it. So my genuinely smart lights (I had them automated to a schedule via a web-based configuration utility, cronkeep) didn't work with my "smart" speaker. Bummer.

Stage three: node-RED!

After a bunch of digging (and finding the IFTTT doesn't work for basically anything anymore) I happened upon the wonders of node-RED. I had a brief run-in with a cloud automation provider, but I won't mention that here. Turns out, node-RED was installed on my Pi all along, waiting for me to add the right Google Home plugin and get to integrating!

It's an integration platform, which let's you connect a vast range of tech stuff to other stuff. Although I did get things working in the end, it wasn't easy. As I said, I'm good with tech, programming, etc. I can generally make stuff work the way I want. It took me: approx. 10-12 hours, 3 plugins, and 4 attempts to get it all working.

And this is mostly Google's fault. You have to register your Google account as a developer account, set up a project for Google Assistant (not Home!?) and configure it properly, add a FireBase database to it, and then 'link' to it on the Google Home app, and then... it works. The LED strip appeared as a light, as configured, and it turned on and off, and via voice commands. And if I turned it on and off via my many other methods, the status in the Google Home app would update immediately and appropriately.

But if this is what it takes me, a techie person, a lot of time and effort to do, how on earth is a regular user expected to do this? Frankly, it's unnacceptable, and offputting for potential users. Not cool, Google, not cool.

Stage four: Sonoff

Once I got the Google Nest mini, I had already resolved to migrate away from using the Pi for the LEDs, for a few reasons. One, the Pi has a limit to the amount of power it can deliver. Two, the Pi got quite hot when running the strip, due to the aforementioned power. And three, I couldn't use the USB ports without also having the lights on. Or if I couldn't turn the lights on, for some reason (e.g. no network) I also couldn't access and use the Pi to fix the issue.

Thus, to remedy this, I ordered a USB-USB device, that is a fancy Wi-Fi switch. It claimed voice assistant support, and I verified that their companion app was in the list of connectable services before buying.

Once it arrived, it wasn't all rosey yet. To get started, I now had to download and install their companion app. And set up an account. Ugh. And then set up the device (oh come on already). I got it a bit wrong the first time, so I reset it and did it right the second time (it was a lot less intuitive than Google's setup process). Then I could go to the Google Home app, over to services, connect to the 3rd party service by signing in with that account I just created, and at last! It works as a first party device.

But that was a terrible experience, one that a regular user would consider a harrowing ordeal, if they even survived managed to complete it. Oh and my widget no longer works, unless I get node-RED involved again (via yet another plugin).

I want something better than that. I know we can do better. We should want to do better. Let's talk about what I think it should be like...

The near-future? experience

Like I said earlier, screw accounts. You're you. Everyone knows that. People don't need you explicitly state your identity and authorize it every time you meet. People recognize other people as who they are. It's highly counter-productive to build tech that enforces principles that are easy for it to understand. Tech should make the effort to work with and for us. Tech should work like us.

You, yourself, are all the authentication and verification a device should ever need. Digital profile? Nah, go make it yourself. Why do I need to tell LinkedIn where I got my education from? My phone should have told it where I went every day for a couple years by now. Profile picture? Grab it from uni, or a photo of me that I've got on my PC (or Google Drive, or my ownCloud). It's all there. Gmail knows where I work. Why doesn't LinkedIn? My laptop knows I'm me, using it, thanks to Windows Hello (fingerprint). Why doesn't the websites I visit know that? Why do they need a separate account? I've already got one for Windows...

Now, I'm not advocating an overarching global conglomerate monopoly or something crazy. I just want it to work together. Totally and completely interconnected. There's a good example of this in the Ender's Game book series: Ender's brother and sister get involved in some political discussion on the Net. Not Reddit, not Twitter, not Tumblr, just 'the Net.' Using what profile? A (illegal) 'adult' profile. Not a Reddit profile where you lied about your age. Their devices know who they are, and thus their real age. Sure, they got around this, but they still only get a 'profile.' Singular. What device? Irrelevant, it's not mentioned, because it doesn't matter. Is it a tablet? A laptop? A phone? It doesn't matter, because in that world, a device is just a device. It has a profile. And that profile works for all the Net.

Imagine just having one profile, accross your phone, laptop, any PC you walk up to, every website, and service. No passwords or authentication to get in the way. Just you, interacting as you want. You can kinda get this, if you use Windows with a Microsoft account and OneDrive (or Apple stuff), but it still doesn't quite work right. Aside from being a reasonably closed "ecosystem," it's still something, and it's not even close to being all-encompassing.

I want nothing. No profiles. No authentication. No confirmation (unless a person would ask for it). Just let me be me, and interact the way I would with anyone else.

This can't be accomplished by any one company, because it's not about expanding your ecosystem to everything you can. That's just a monopoly, and doesn't help the situation, because it's still exclusive. This is about breaking the "ecosystem". Such a concept shouldn't exist anymore.

In Star Trek data communication is effortless. Captains can promise information will be delivered, and it will. Totally different technology systems, totally different ships, differrent species? Irrelevant, it's all interoperable at some level, and that's enough. Thousand-year-old alien ship? Compatible, but never before seen. That's the level of connectedness I wish we wanted.

"Ooh but what about privacy?"

Who said any of this information needed to be shared? You can keep it private still. Just as friends are discretionary about sharing your personal information, so can tech. If a calendar service wants a look through your pictures, it better have a reason. It wants a profile pic? Well send it your profile pic, that's public anyway. It wants contact photos? No, let it link to the contacts, it shouldn't need to touch their details.

Also, all this stuff can be processed locally. If you're setting up a new device, it might ask who its primary user is. Naturally, you'd answer, "I am." So it'd probably ask, "And how should I address you?" And you give it your name. Now it's got a basic voice print, and it knows your name (and it's physically in a location). So now, it can inquire about other devices around it if they already know you. They do? Success, it's now a part of your tech, and it works together. Any other information it needs can be asked of the other devices that already know it.

Or are you in public? In that case, it should interrogate the devices that you're carrying about who you are. This is where that Iron Man tracking tech comes in. Oops, you're not carrying any tech with you right now. Well that's okay, go check with the government (equivalent to checking your driving licence as valid ID), and see if they've got the required information. If not, grab your current address, and phone home. Surely you've got something at home that knows something.

If all that fails, then, and only then, should it bother you with pertinent queries. Ideally this would happen in realtime, in the background, so the user never experiences a delay, except when it is unavoidable. If there's some time to wait, the device should move on if possible with something else, or just inform you that it'll take, "one moment, please" or, "just let me check something first."

Alright, I'm done

I hope that wasn't too much of a wall of text to read. Anyway, there's probably another post that's an easier read to refresh yourself with. Thanks for sticking it through this one anyway.

If there's any niggling little things left, please see the home page for contact information.