Friday, January 29, 2021

Time is of the essence

For an index to all my stories click this text

I am building a project together with my son in law (he does the mechanics) in which we need to know the time to set some lamps on or off. The project uses an ESP32 as we need a lot of IO ports.

How do we get the time. Of course there is a timer that starts everytime the ESP boots. But that does not give you the actual time. And that is what we need. Lamps should go on when it gets dark and not in bright sunlight.

Luckily there is a library for the ESP micro controllers that will get a fair accuarte time from an official NTP server. Fair accurate ?? Well yes it takes some time to get the information from the server (somewhere on the world) to the ESP and that can set the time one or two seconds off. Nothing we need worry about as our project is not THAT critical.

Libraries.

To get the time the ESP program needs several libraries. First it needs to make contact over the internet with the Wifi library which is included in the ESP setup in the Arduino IDE.

Then we need the NTP library to make contact over the web with an NTP server. These are very accurate servers that have only one purpose: serve the actual time. The library will get the time from the server.
This library is suitable for the ESP8266 (all versions) and the ESP32.

Next we need the UDP library as the NTP server sends the information in UDP protocol. This is also included in the Arduino IDE.

So we only need to install the NTP library to get things working. The others are already present.



To get the library just open the library manager in the Arduino IDE



Search for NTP and choose the NTP Client by Fabrice Weinberg.

The time program

The source code is for the ESP8266 and you will need to change just 1 line to get this working on an ESP32. I'll show you how to do that at the end of the explanation of the program.


#include <NTPClient.h>
#include <ESP8266WiFi.h>
#include <WiFiUDP.h>
 
// WiFi credentials for your router
const char* ssid = "XXXXXXXXXXXXXXXXX"; 
const char* password = "YYYYYYYYYYYYYYY"; 

#define NTP_OFFSET   60 * 60      // In seconds
#define NTP_ADDRESS  "europe.pool.ntp.org"

WiFiUDP ntpUDP;

NTPClient timeClient(ntpUDP, NTP_ADDRESS, NTP_OFFSET);

void setup(void) 
{  
  Serial.begin(115200);
  Serial.print("Starting the connection");
  WiFi.begin(ssid, password); // Connect to WiFi
  
  timeClient.begin();
  
  while (WiFi.status() != WL_CONNECTED) 
  {
    delay(500);
    Serial.print(".");
  }
  // Print local IP address and start web server
  Serial.println("");
  Serial.println("WiFi connected.");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}
 
void loop() {
  timeClient.update();
  String newtime = timeClient.getFormattedTime();
  Serial.print("the time is : ");
  Serial.println(newtime);
  Serial.print("Hour    : ");
  Serial.println(newtime.substring(0,2));
  Serial.print("Minute  : ");
  Serial.println(newtime.substring(3,5));
  Serial.print("Seconds : ");
  Serial.println(newtime.substring(6,8));
  delay(1000*10);
}

I will go over the program step by step. Thats the best way to comprehend and alter it for your own purposes.


#include <NTPClient.h>
#include <ESP8266WiFi.h>
#include <WiFiUDP.h>
The program starts with loading the libraries it needs.


// WiFi credentials for your router
const char* ssid = "XXXXXXXXXXXXXXXXX"; 
const char* password = "YYYYYYYYYYYYYY"; 
This is all familiar stuff. Replace the XXXXXX with the name of your router and the YYYY with it's password.


#define NTP_OFFSET   60 * 60      // In seconds
#define NTP_ADDRESS  "europe.pool.ntp.org"
The NTP_OFFSET defines in which time-zone you are. I am in the Netherlands at 1 hour after the Greenwhich time. So the offset for me is 60 seconds x 60 minutes.
The NTP_ADDRESS is the name of the NTP server. I am using the European version.


WiFiUDP ntpUDP;

This line starts the UDP library.


WiFiUDP ntpUDP;

And here the NTP library is started with UDP, the servers adress and the time-zone offset.

The setup() will bring nothing new. The serial monitor is started. It starts the Wifi connection and as long as there is no connection it prints dots. And when a connection is established it prints the IP adress in the Serial monitor.

The loop is where we extract the actual time.


  timeClient.update();
  String newtime = timeClient.getFormattedTime();

The first line gets the new time from the server and the second line puts that time in the  String variable newtime. We can analyse that variable to extract hours, minutes and seconds.



As you can see the time is printed in the serial monitor in the format HH:MM:SS.


  Serial.print("the time is : ");
  Serial.println(newtime);

These lines print the time in the Serial Monitor.

  Serial.print("Hour    : ");
  Serial.println(newtime.substring(0,2));
  Serial.print("Minute  : ");
  Serial.println(newtime.substring(3,5));
  Serial.print("Seconds : ");
  Serial.println(newtime.substring(6,8));

This is where the hours, minutes and seconds are extracted. A string starts at index 0 so substring(0,2) gets the first two letters in the string which are the hours. substring (3,5) Gets the characters at the 4th and 5th position which are the minutes.

The program then waits 10 seconds and gets the new time from the server.

The ESP32 version

Well actually the program is exactly the same except for the call to the wifi library. The only thing you have to do is to alter:

#include <ESP8266WiFi.h>

into:

#include <WiFi.h>

The NTP server

As I am located in the Netherlands I use the european NTP server located at: europe.pool.ntp.org

There are many more servers and at least one for each continent. You can find them all on the following link: https://www.ntppool.org/zone/@
Please use a NTP server closest to your location for more accuracy.

Time zones

The NTP servers send UTC time. You might be in a different timezone. To get the right time on your screen use the following formula:

UTC -5.00 : -5 * 60 * 60 : -18000
UTC +1.00 : 1 * 60 * 60 : 3600
UTC +0.00 : 0 * 60 * 60 : 0

#define NTP_OFFSET   60 * 60

This is my adaption as I am located at UTC +1 hour.

Now we can accurately set our lights on and off depending on the hour of the day.


BAD PRACTICE

The program demonstrated here is actually bad practice. You are overloading the NTP server with your requests. What you should do is to request the time once a day or every 6 hours or so. This way we can keep the NTP servers free of charge. 

To keep the time you should use the time library and have that updated every once in a while with the accurate time from the NTP server. That is for another story.


Till next time
Have fun

Luc Volders

Friday, January 22, 2021

Control Domoticz with your own APP

For an index to all my stories click this text

 
As you might know by now I am using the Domoticz Software for my home automation. For switching the lights in my home I am using 433Mhz lightswitches from KlikAanKlikUit and mostly unknown brands from the dollar shop. These are controlled by their own Remote Controls AND by Domoticz.


Domoticz is great. It runs on a Raspberry and can be controlled from a Dashboard that is accessible from the Raspberry itself and/or remote from a PC. Next to that you can still use the original Remote controls that are supplied with the wireless light switches. There also is an App for Android and IOS so you even check the status of several appliances and regulate your lights from your work or holliday. Check it out: https://www.domoticz.com/

And herein lies the (small) problem is was facing.
I have several lamps that are controlled by remote control in my living room. However most remote controls are only capable to switch 4 appliences. And I have more. This a problem for my girlfriend who has to use multiple remote controls just to switch lamps on or off. As she is computer illiterate I wanted a simple way to control these lamps. So why not use the Domoticz app. Well I do not want her to accidentally fiddle with the settings. So that is out of the question.

MIT's App Inventor to the rescue !!

So why not make my own APP for Domoticz. It's not that complicated. I used the unsurpassed Mit's App Inventor to build this APP for Android. You can find APP-Inventor here: http://appinventor.mit.edu/explore/



Above is my layout. I am, so this app also is, Dutch. However you can easily exchange the names of the buttons to your own liking.

I am not going to give you a full tutorial on App-Inventor here as there are many tutorials to be found on the internet and there is even a complete course App-Devellopment at EDX : https://www.edx.org/course?search_query=app+inventor

So I'll give you the outline here on how I build my App.


In the Designer window at the top put a label. I gave it the name woonkamer which means living room. However give it any name you like.

Next put a table arrangement on the screen. In this we are going to put our buttons with some spacing between them. So I made it 3 columns wide and 16 rows high. You can expand it as much as you like but then make sure to make the screen scrollable.

In each left column I put an ON (green colored) button and on the right side an OFF (red colored) button. In the middle of the first column I put a label in which I only put a space. This makes sure that there is some horizontal space between the buttons.
Below each row I put a label. This makes sure there is some space vertically between the buttons.

My screen background is pinkish you can alter that to your own liking in the Screen properties.

On the bottom of the screen there is one hidden component called Web1. You can find that in the connectivity tab on the left side of App-Inventors designer screen. This component makes sure we can communicatie over the internet.

The last 2 ON and OFF buttons are called "Niet in gebruik" and are for future expansion. I'll be using them for the lights of my Christmass Tree.

That's about it for the designer screen.

Now let's have a look at the program.

Blocks

We have 14 buttons in total. Seven ON buttons and 7 OFF buttons. Basically they are the same and the only difference is the ID-number that Domoticz has given the Switch and wether it is ON or OFF.


Now this how the blocks for BUTTON 1 and BUTTON 2 look. The first one sets the light "Vensterbank" ON and the second puts it off.

The set WEB1 command is not completely shown in the above screendump. The actual text is:

http://XXX.XXX.XXX.XXX:8081/json.htm?username=YYYYY=&password=ZZZZZ=&type=command&param=switchlight&idx=1446&switchcmd=On

As you can see we have to fill in several parameters. The command needs to know the IP adress where to send the command to. The username and password of your Domoticz system.

IP Adress.

There are two different IP adresses that you can fill in here. The first is your IP adress from within your own network. Your local IP adress. You'll use that one if you are not going to use this APP from outside your home.
If you are going to use the APP from within your own network you can leave phrases for the username and password out.

You can find your local IP adress within your router.



I have a Zyxell router and when I open it's webpage I can see all my devices. At the top are the Wifi devices. At the bottom the wired devices and there you can see Domotcz. Clicking on it will reveal it's IP adress.

If you are using the APP from outside your house you will need to fill in the external IP adress.

You can find your IP adress on this website: http://www.whatsmyip.org/

It will display the following information:



Next to that you will have to open a port in your router which allows the APP to communicate over the internet to Domoticz.

Dig into the setting pages from your router.
Look at something called port-forwarding.

There you can open a new port for the outside world.

In my router I have to give it a name. Next I give it the IP adress of Domoticz and Domoticz port number which is 8080.
Next I instruct it to send all communication from router port no 8081 to the Domoticz port.



The router will most likely ask for a starting port number and an ending port number for your local and global ports. In both cases use the same. So start local 8080, end local 8080, start global 8081 and end global 8081. You can use many numbers as a port number but there are some restrictions. So delve into that by checkin information on the internet about this. This method allows you to open many specific ports for all kinds of projects. We've done that before on this weblog to open up communication to an ESP from anywhere in the world: https://lucstechblog.blogspot.com/2017/05/google-home-and-esp8266.html


Next step is to fill in your username and password.
First you need to set them in the Domoticz setting page. My Domoticz system is in Dutch but look under settings and where I pointed the arrows to.

When you have set a username and password we need to fill these in in the App's blocks.
You can not just fill in the name and password you just typed into Domoticz. Domoticz has a safety feature and that is that it uses base 64 Encode.

So you'll have to encode your username and password with base 64.
You can do that using this link: https://codebeautify.org/base64-encode
Check this site. it has many usefull features and conversions.

Almost there.

The last thing we need to know is the ID of the switch we are going to control.




In Domoticz go to the settings page and look for Apparatus. Now choose only the used apparatus. And there we have all our switches with their IDX.

All data available

Now we have all data available we can fill it in into the WEB 1 Command:

http://XXX.XXX.XXX.XXX:8081/json.htm?username=YYYYY=&password=ZZZZZ=&type=command&param=switchlight&idx=1446&switchcmd=On

The above command is for Button 1 (ON) (Vensterbank with ID 1446) and the next is almost the same for Button 2 (OFF)

http://XXX.XXX.XXX.XXX:8081/json.htm?username=YYYYY=&password=ZZZZZ=&type=command&param=switchlight&idx=1446&switchcmd=Off

Repeat this for all your buttons and replace the ID numbers with the ones you find in your Domoticz settings.

Done !!
Your app is finished. Test it with App Inventors AI companion and make sure all functions work as they should.

Now transfer you APP to your Phone/Tablet and start controlling your home from anywhere in the world with an easy app.

Till next time
Have fun

Luc Volders

Friday, January 8, 2021

ESP8266 Ticker library problem

For a complete index of all my stories click this text

I ran into a problem. It took me several test and a lot of time to discover what was wrong.
Let me first tell you what I was trying to achieve.

I was building a webpage with an ESP8266 using the Arduino IDE. The webpage had some buttons and displayed the temperature from an Dallas DS18B20. Next to that I wanted to send the readings also to Domoticz. Nothing dramatic.

The problem rose when I wanted to automatically update the temperature reading on the webpage every x minutes.  I used the Ticker library for that.

Ticker

Ticker is a build-in library for the ESP8266 which allows you to have a process run automatically every x seconds or x milliseconds.
The use of it is very simple. I'll show you an example that switches a led ON and OFF every second in the background while in the meantime the program can do all kind of other things.


#include <Ticker.h>

Ticker everytime;
int Ledstatus = 1;

void setup()
{
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);
  everytime.attach(1, blinkled);
}

void loop()
{ 
  // put your code here
}

void blinkled()
{
  Ledstatus = 1 - Ledstatus;
  digitalWrite(LED_BUILTIN, Ledstatus);
}

The program looks simple and indeed it is.


#include <Ticker.h>

This loads the Ticker library when the program is compiled.

Ticker everytime;

This line glues the Ticker library to our name everytime.

In the setup the build-in ledpin of the Wemos D1 Mini is defined as an output and the led is put OFF (HIGH). Looks strange to put a led OFF by setting the pin HIGH but remember that the pins of the Wemos D1 mini and NodeMCU build-in led are reversed.

everytime.attach(1, blinkled);

This line in the setup sends the routine eveyr second (1) to the blinkled subroutine. Change the 1 in any number of seconds you'd like. Or change attach in attach_ms for miliseconds.

The blinkled routine just puts the led ON or OFF and speaks for itself.

Does this work. ????
Yes it works flawlessly till ...............
Till you start to use wifi.

As soon as you use wifi it is blocked by the Ticker library. So my Dallas temperature routine worked. My Webpage page worked and detected button presses and send them to Domoticz. And as soon as I invoked Ticker my webservice stopped working.

The solution.

Back to the old Arduino Blink without delay routine !!

I adapted this routine for my purposes and then everything worked flawlessly.


// Blink without delay
// adapted by Luc Volders

int Ledstatus = 1;

unsigned long oldMillis = 0;
const long blinkrate = 1000;

void setup() 
{
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop() 
{
  unsigned long newMillis = millis();
  if (newMillis - oldMillis >= blinkrate) 
  {
  oldMillis = newMillis;
  Ledstatus = 1 - Ledstatus;
  digitalWrite(LED_BUILTIN, Ledstatus);
  }
}


Just a quick examination.

unsigned long oldMillis = 0;
const long blinkrate = 1000;

The variables are initialised. Alter the blinkrate value to your needs. I used 1000 in this test. That is 1000 Miliseconds and equals 1 second. If you'd need something to happen every 10 minutes it would be 1000 (millis) x 60 (seconds) x 10 minutes = 600000

And for the rest everything is done in the loop.

unsigned long newMillis = millis();

First we test how many Milliseconds have past.

if (newMillis - oldMillis >= blinkrate)

And then we test if the amount that has past minus the amount when we started is larger as the 1000 we set as a tresh-hold (1 second). If that is true:

  oldMillis = newMillis;
  Ledstatus = 1 - Ledstatus;
  digitalWrite(LED_BUILTIN, Ledstatus);

The value of the time past until that moment is put into oldMillis and the leds pin value is reversed. The counter then starts anew.

This works as expected and does not have any influence on the wifi operation.

So no problem here ??

Whish that was true. Life is not that simple. There is a problem with this setup, and how severe it is depends on the kind of program you are writing.

Suppose there is a lot going on in your program and from the loop you call all kinds of functions. If these functions take a lot of time it might be longer as 1 second before the loop is finished and starts again.

Then the time that has elapsed is longer as 1000 millis and your led might blink irreuglarly.

Now when you are timing something that only happens once every 10 minutes it is not a big problem. However if your program is time-critical you might get unexpected results. You could adapt the interval value to straighten that out.

Fortunately there are some other libraries that do not have these problems. I am testing some and will get back on this.

Experiment and have fun !!!
Till next time

Luc