Getting started with embedded development can be difficult and expensive, but in this article I will show you how to spin up FreeRTOS and the lwIP TCP/IP stack on a WiFi chip with a secure embedded web server in less than 30 minutes on a super low cost device.
Why use a Secure (TLS) Enabled Server?
Browsers have started flagging standard (HTTP) servers as “Not trusted” in the browser bar (see Figure 2 below), and a secure server will appear more trustworthy to the end customer. I'll go into this in detail in the Security Considerations section below.
The ESP8266 WiFi Chip and the Minnow Server
You can get ESP8266 for as low as $1, but I recommend using a board that includes USB. The USB is needed for flashing the firmware. An additional step that includes using a USB to TTL converter would be needed without onboard USB. A ready to use ESP8266 board with USB can be purchased for as low as $3.
The low cost ESP8266 WiFi chip is great for learning purposes , but it is slow, especially with CPU intensive tasks such as performing encryption. You really need to think about the design when using a slow chip with CPU intensive tasks.
The ESP8266 runs at 80 Mhz which is good for an embedded chip, but the code execution is dog slow with the code executed via SPI memory. Running a TLS enabled server on this device provides a real wakeup call when it comes to practical limitations with resource constrained microcontrollers and provides a great introduction to what is possible to run on a slow microcontroller. The type of secure web server chosen and web application design becomes critical for slow devices.
In this article, I will explain how to compile and run the Minnow Server on an ESP8266. The Minnow Server and the reference example is available on GitHub.
I recently published an article on the Minnow Server here at embedded.com. The article explains how to use the Minnow Server for creating modern web based device management applications. The article also goes into the benefits of using a WebSocket server with a so called Single Page Application (SPA) when using TLS. I recommend reading this article since we will be preparing and compiling the same example code for the ESP8266.
We will be using the ESP8266 FreeRTOS/lwIP environment provided by the esp-open-rtos project, but don't worry, you do not need to go through the complex esp-open-rtos setup. Instead, we will be using a pre-configured and completely ready to use environment with an easy to use web-based IDE.
Download the FreeRTOS/lwIP ESP8266 IDE
Navigate to Real Time Logic and download the ESP8266 IDE we have prepared. The IDE, which is designed for educational purposes, includes a pre-configured esp-open-rtos bundled with an easy to use web-based C source code IDE. The IDE requires VmWare or VirtualBox. I prefer VmWare over VirtualBox, since I find it easier to use. Note that VmWare is free for non-commercial use.
The ESP8266 IDE also includes an embedded TLS stack (SharkSSL), and the Minnow Server automatically uses TLS when compiled with SharkSSL.
You can start using the ESP8266 IDE without the ESP8266 board, but you cannot upload and run the compiled code. In other words, you can download the IDE and follow all steps in this tutorial except for running the code on the ESP8266.
The screenshot in Figure 1 below shows the virtual machine and the web-based IDE connected to the virtual machine. Notice how the virtual machine has taken ownership of the ESP8266 USB connection. The web-based IDE detects this and shows the ESP8266 as connected and ready to be flashed with new firmware.
click for larger image
Figure 1: Web based IDE connected to the IP address of the virtual machine. (Source: Real Time Logic)
The screenshot in Figure 1 shows the ESP8266 connected to LEDs via a breadboard. You do not need the extra LEDS or the breadboard unless you want to control the external LEDS used by the example application.
Note that the IP address of your VM will most likely be different than in Figure 1. Make sure to click in the virtual machine window for the IP address. If the window is blank, click the window and then press the enter key. You must enter this IP address in your browser to open the web IDE.
Installing the Minnow Server and the Example Code
The Minnow Server reference example is not included in the ESP8266 IDE and must be installed separately. Figure 1 shows the Minnow Server example already installed in the 'ESP/ms' directory. The Minnow Server must be installed from the Linux command line, but don't worry if you have not used Linux. You can simply copy and paste the commands we have prepared. The following figure shows a screenshot of the Minnow Server installation commands being pasted into the Linux web shell available at http://vm-ip-address/webshell/.
click for larger image
Figure 2: Minnow Server installation commands pasted into Linux web shell. (Source: Real Time Logic)
When you enter the URL to the web shell in your browser, you will be prompted to log in. The username is sharkssl and the password is SharkSSL.
The ESP8266 tutorial on GitHub explains how to configure the virtual machine, install everything, compile the web server example, and flash the firmware. The following video simplifies the process by explaining everything step by step.
Video 1: How to download and install the Minnow Server example, compile, and upload the code. (Source: Real Time Logic)
Secure vs. Non Secure
The Minnow Server can be used in secure or non-secure mode and automatically uses secure mode (TLS) when used with the ESP8266 IDE and SharkSSL. Using a TLS enabled web server in a device adds additional complexity, and the end user gets an administrative burden of dealing with SSL certificates. In the above video, we show how to remove the certificate error presented in the browser by installing the Certificate Authority (CA) certificate in the browser. The CA certificate is the certificate that was used to sign the certificate in the web server example.
Embedded devices are typically installed in protected private networks, but many customers still demand that the web server connections use TLS. The use of TLS enabled servers on private networks is problematic since no well-known certificate authority will sign IP addresses or non-standard (private network) names. One option is to be your own certificate authority. If you are interested in learning more about this, check out our tutorial How to act as a Certificate Authority.
As mentioned above, the ESP8266 is slow. TLS is very CPU intensive and you will notice that it takes some time to connect and establish a connection. The Single Page Application example, which uses WebSockets for communication, is optimized for CPU constrained devices, but the initial asymmetric encryption (TLS handshake) still takes time, especially with some browsers that do not follow good design practices and open many TCP connections that all require full TLS handshake. To learn more about the problem with modern web browsers and slow microcontrollers, see our tutorial Creating Single-Page Apps with the Minnow Server, which goes into the problem in details.
Developer Mode vs. Release Mode
In the above video, we are compiling for release mode by embedding the Single Page Application (HTML resources) in the firmware. That way, we can simply navigate to the device to start the application which gets loaded on demand into the browser.
Note that the server side (device side) for the Single Page Application provides a porting API to the ESP8266. This API works with the drag and drop firmware upload example, but the uploaded data is not saved. Also, changing credentials is not implemented. The esp-open-rtos API includes a file system and you may add the missing functionality by using the APIs provided by the esp-open-rtos.
IoT Enabling the ESP8266 Reference Example
Devices with an embedded web server are typically installed and operated on private networks behind company firewalls and/or routers and are automatically protected from outside use and abuse by the firewall. However, such deployments also prevent legitimate remote use.
Sometimes, embedded devices need to be operated from a remote location via the Internet, and the safest way to do this is to IoT enable the device and have the device connect to a server on the Internet. The following figure depicts such a setup.
Figure 3: Access the embedded device locally or remotely via the Internet. (Source: Real Time Logic)
The Minnow Server reference example includes an optional IoT component that can be enabled at compile time. Note that this component is disabled by default and requires that you make some minor modifications to the code (or to the Makefile) to enable it. The second part of the above video shows how to connect to an online test server we have setup.
When the device runs in IoT mode, the device operates as a network client in addition to acting as a server for local use. Network clients typically do not need any network configuration for connecting out to services on the Internet, but a network client must connect to a known service. For this reason, we have set up an online IoT server that you may use for test purposes.
The online server acts as a proxy between the remote users and the devices and lets remote users securely access private web servers running behind a firewall. Note that the test server we have setup is not a ready to use service. We will keep it running for a while, but eventually you will need to setup your own private IoT server if you plan on using the IoT mode.
The GitHub Minnow Page includes instructions on how to setup your own private IoT server. The benefit with a privately owned IoT server is that you are not dependent on any particular IoT service provider. You will be surprised to find out how easy and reasonable it is to operate a private IoT server.
How to Enable IoT Mode
The above video shows how to IoT enable the Minnow Server reference example and the following screenshot from the video shows the macros that need to be enabled in the Makefile.
click for larger image
Figure 4: macro 'USE_SMQ' enables IoT mode and macro 'SMQ_DOMAIN' sets the server name. (Source: Real Time Logic)
As you can see from Figure 4, the SMQ_DOMAIN macro must be set to our online IoT demo server “minnow.ml” or to your own IoT server.
At this point, you should be able to click the Run button, which compiles and uploads the new firmware. When the new firmware runs, you should see the following text being printed in the web console. Note that I have removed some of the printouts.
The reason for the certificate warning has to do with the device which now acts as a TLS client not trusting the online server. Certificate management is complex and I'll get into this at the end of the article for the once interested.
All connected devices show up in the online server's (minnow.ml) user interface. Clicking one of the devices takes you to the device's login screen. The login page is for the credentials provided by the device and not the online server. The default credentials are root/password. Figure 5 shows the online server before and after clicking a device link.
click for larger image
Figure 5: Online server before and after clicking a device link. (Source: Real Time Logic)
The left pane of Figure 5 shows one device connected, however the online server can manage any number of devices and the list grows as more devices connect. When you click a link, the online server's proxy functionality connects you to the device, and from this point on, the connection conceptually works just like the direct WebSocket connection shown in Figure 3.
As mentioned above, the Minnow Server can operate in either secure (HTTPS) or non secure (HTTP) mode. Using a non secure connection is much easier since you do not have to deal with certificates. The server is also considerably faster on slow CPUs. However, as shown in the browser bar in Figure 2 above, Google's Chrome browser and other browser vendors now show a warning message when the connection is not secure. Who knows what they will do next? Google seems determined to want to axe non-secure communication.
When using TLS you must make sure you use a valid chain of trust. All browsers throw up a big warning message if a user browses to a server with a non-trusted certificate. As we explained above and showed in the video above, setting up a chain of trust for private servers requires that you come up with a solution for trust management on a private network. One solution, as shown in the above video, is to act as your own certificate authority. I recommend reading the article How to act as a Certificate Authority if you are interested in setting up your own chain of trust.
When the device operates in client mode and connects to the online test server minnow.ml, a certificate warning is printed in the console. You can find this printout in the example source code file “main.c” (See Figure 3). Instead of printing a warning, the connection should really be terminated, but before modifying the C code to terminate non-trusted connections, you must make sure the device trusts the online server minnow.ml. Trust can be restored by importing the CA certificate (the root certificate) for the online server into the device's C code. The online server's certificate is signed by Let's Encrypt.
If you are new to the concept “chain of trust”, I recommend reading the article Certificate Management for Embedded Systems.
That's about it. Should you have any questions, please ask in the comments section below.
Wilfred Nilsen, Founder & CTO of Real Time Logic, has 28 years' experience in designing embedded software. Powered by a vision of connected embedded systems, he designed the Barracuda Application Server, tailoring it for the small footprint, real-time needs of embedded microprocessors.