The Ayla ESP32 solution includes example host application source code, an Ayla Wi-Fi Integrated Agent, and Ayla extensions to the Espressif IoT Development Framework (IDF). These pieces link together to provide an Ayla edge solution for an ESP32 or ESP32-C3 module. Learn more about Ayla Integrated Agents.

First steps

Obtain an Espressif dev kit. Any of the following would work: ESP32-DevKitC,
ESP32-C3-DevKitM-1,
ESP32-C3-DevKitC-02.
Create an Ayla user account.
If you do not have an account already, please contact Ayla Support team to get user account with admin or developer access rights.
Reserve a DSN.
To reserve a DSN, follow these steps:
1. Sign in to the Ayla Customer Dashboard.
2. On the left navigation pane, click Factory Actions.
3. Click the Reserve DSN button.
4. For Model, select AY008ESP1.
5. Click Submit and then download the associated XML file. You will need the DSN value and key from the downloaded file.
NOTE: If you do not have access to the dashboard, request a DSN from your Solution Engineer or from Ayla Support team.
Sign in to the Ayla Developer Portal and create a template with the details provided in the following table. Note that the template version is different from the ADA version.
The OEM model describes the application and platform, it could be demo-esp32 for the ESP32-WROOM-32E module, and demo-esp32c3 for the ESP32-C3 modules.

Visibility: private

Name: demo-esp32-msgs 1.10

Description: Demo ESP32 msgs 1.10

Registration: AP-Mode

Model: demo-esp32 or demo-esp32c3

Version: demo_esp_msg 1.10

Type: Wifi

To learn how to create templates, please refer to Ayla Developer Portal User Guide.
Select the new template and click the Properties tab.

Copy and paste the following properties into a text file (e.g. properties.csv), and import the file using the Import button.

base_type,direction,name,scope,mime_type,ack_enabled,host_sw_version,track_only_changes
message,input,binary_in,user,application/octet-stream,TRUE,FALSE,FALSE
message,output,binary_out,user,application/octet-stream,FALSE,FALSE,FALSE
boolean,output,Blue_button,user,text/plain,FALSE,FALSE,FALSE
boolean,input,Blue_LED,user,text/plain,FALSE,FALSE,FALSE
string,input,cmd,user,text/plain,FALSE,FALSE,FALSE
decimal,input,decimal_in,user,text/plain,FALSE,FALSE,FALSE
decimal,output,decimal_out,user,text/plain,FALSE,FALSE,FALSE
boolean,input,Green_LED,user,text/plain,FALSE,FALSE,FALSE
integer,input,input,user,null,TRUE,FALSE,FALSE
message,input,json_in,user,application/json,TRUE,FALSE,FALSE
message,output,json_out,user,application/json,FALSE,FALSE,FALSE
string,output,log,user,text/plain,FALSE,FALSE,FALSE
string,input,message_start,user,null,FALSE,FALSE,FALSE
boolean,input,node_batch_hold,user,null,FALSE,FALSE,FALSE
integer,output,output,user,text/plain,FALSE,FALSE,FALSE
string,output,schedule_out,user,text/plain,FALSE,FALSE,FALSE
file,input,stream_down,user,text/plain,FALSE,FALSE,FALSE
integer,output,stream_down_len,user,text/plain,FALSE,FALSE,FALSE
integer,output,stream_down_match_len,user,text/plain,FALSE,FALSE,FALSE
file,output,stream_up,user,text/plain,FALSE,FALSE,FALSE
integer,input,stream_up_len,user,text/plain,FALSE,FALSE,FALSE
message,input,string_in,user,text/plain,TRUE,FALSE,FALSE
message,output,string_out,user,text/plain,FALSE,FALSE,FALSE
string,output,version,user,null,FALSE,TRUE,TRUE

Add schedules to the template. The names should be "sched1" through "sched5", and the
direction should be to-device.
Select the SUNRISE/SUNSET SUPPORTED option for the schedules.
Under Available Connects, make sure that the LAN Connect and Auto-sync options are selected.
Do not select BLE Connect unless you will be using the BLE Local Control feature, as it requires additional memory.

Build and Run

GCC/Ubuntu/Docker

This section provides directions for building an Ayla ESP32 Solution v1.11 using a GCC/Ubuntu/Docker development environment. The recommendation is to use Ubuntu version 20.04 tls. Running Ubuntu natively or in a VM is supported.

Download Ayla source code (ada-esp-idf-src-1.11.tgz).
ada stands for Ayla Device Agent.
NOTE: If you do not have access to Ayla source code, please request for the access to Ayla Connection from Ayla Support team.
Determine the computer-to-ESP32 serial port:
1. List the serial ports on your computer (e.g. ls /dev/tty*).
2. Connect an ESP32 board to your computer using a serial cable.
3. List the serial ports again. The new serial port on the list is the one to use (e.g. /dev/ttyUSB0).
Install a Docker Engine on your Windows, Mac, and/or Linux host OS.
Create a Docker 64-bit Ubuntu container.

$ docker run --net=host --name=ada17 -w /root -it --device=/dev/ttyUSB0 ubuntu bash

You are user root, and the current working directory is /root. To exit your Docker container, type exit. To restart and re-attach to your Docker container, type the following:
$ docker ps -a
$ docker start ada17
$ docker attach ada17

Prepare your development environment:
1. Update the package list in your Ubuntu Docker instance:
```
# apt update
```
2. Install miscellaneous packages:
```
# apt install nano screen iputils-ping
```
3. Install prerequisites for Espressif Standard Setup of Toolchain v4.3.2 for Linux.
```
# apt install gcc git wget make libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-cryptography python-future python-pyparsing
```
  If using Ubuntu 20.04, install using the following command:
```
#  apt install gcc git wget make libncurses-dev flex bison gperf python3 python3-pip python3-setuptools python3-serial python3-cryptography python3-future python3-pyparsing
```
  If using Ubuntu 20.04, link using:
```
# ln -s /usr/bin/python3 /usr/bin/python
```
4. Create an esp directory, and change directory:
```
# mkdir esp
# cd esp
```
5. Clone the Espressif IoT Development Framework:
```
# git clone -b v4.3.2 --recursive https://github.com/espressif/esp-idf.git esp-idf-v4.3.2
```
6. Create the IDF_PATH environment variable, and persist the change in .bashrc:
```
# export IDF_PATH=$HOME/esp/esp-idf-v4.3.2
# echo "export IDF_PATH=\$HOME/esp/esp-idf-v4.3.2" >> ~/.bashrc
```
7. Set up the Espressif tools using the following instructions:
  1. cd $IDF_PATH
  2. export IDF_GITHUB_ASSETS="dl.espressif.com/github_assets"
  3. ./install.sh 'esp32;esp32c3'
  4. . ./export.sh
  5. The above adds to the PATH environment variable. Persist the change in .bashrc:
```
# echo "export PATH=$PATH" >> ~/.bashrc
```

Test the development environment with the hello_world example:

Change directory:

# cd $IDF_PATH/examples/get-started/hello_world

Customize configuration. (Defaults should be fine.)
Set target to either esp32 or esp32c3, depending on your devkit.
# idf.py set-target esp32 # idf.py menuconfig

Build, flash, and monitor:

# idf.py build
# idf.py flash
# idf.py monitor

Output appears similar to this:

Toolchain path: /root/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a
Compiler version: 5.2.0
...
...
Hello world!
This is ESP32 chip with 2 CPU cores, WiFi/BT/BLE, silicon revision 1, 4MB external flash
Restarting in 10 seconds...
Restarting in 9 seconds...
...
...

Press Ctl + ] to exit the monitor.

Install Ayla source code:
1. In a host terminal, copy ada-esp-idf-src-1.11.tgz to your Docker container:
```
$ docker cp /home/matt/Downloads/ada-esp-idf-src-1.11.tgz ada16:/root/esp
```
2. In your Docker terminal, extract the archive file:
```
# cd /root/esp
# tar zxvf ada-esp-idf-src-1.11.tgz
```
3. Copy the ada-esp-idf-src-1.11 directory on top of the esp-idf-v4.3.2 directory:
```
# cp -R ada-esp-idf-src-1.11/* ./esp-idf-v4.3.2
```

Configure and build the Ayla agent and host application:

Change directory:
```
# cd $IDF_PATH/examples/ayla_demo
```

Build, flash, and monitor:

# idf.py set-target esp32 # or esp32c3 as appropriate
# idf.py build
# idf.py erase_flash
# idf.py flash
# idf.py monitor

Provision the device DSN and OEM credentials. Configure the same OEM model used in the template.

    # nvs-set "ada.f.id/dev_id" AC000W000000001
    # nvs-set "ada.f.id/key" MIIB... (long number)
    # nvs-set ada.f.client/server/default 1
    # esp-reboot
    # oem 000000000 (replace with your Ayla OEM ID)
    # oem model demo-esp32    (replace with OEM model from template)
    # oem key 0123456789abcdef0123456789abcdef demo-esp32 (replace with your OEM secret and the OEM model from the template)
    # wifi profile ap ssid Ayla-DevKit
    # setup_mode disable
    # save

NOTE: Setting "ada.f.client/server/default" to 1 will force your devices to use the development servers. For production, please set this value to 0: "ada.f.client/server/default 0".

Register your device

The Aura Mobile App running on iOS or Android communicates with your ESP32 device (via Bluetooth) to add the device to your local network, and then register the device to your Ayla user account. For instructions, see Using BLE to set up Wi-Fi for ESP32.

Sample Application

The sample application provided below contains working examples of how Ayla Embedded Agent APIs may be used. This code should be evaluated as an example only and not to be considered as the only way or best practice when using the APIs.

Properties

The ledevb demo in examples/ayla_demo/main implements a subset of the Ayla development kit demo properties:

Button 0 (which is the Boot button): used for the Blue_Button property as well as the Button Push method of registration. Pressing button 0 for more than 5 seconds starts the registration window.
The Blue_Button property: indicates the current state of button 0, (GPIO_0 on ESP32). 1 indicates pressed, and 0 indicates not pressed. The value is sent on each change.
The decimal_in property to the device is reflected back as the decimal_out property from the device as an example of how to use decimal properties. The value in the demo is 100 times the value entered. For example, if 1.23 is entered on the developer site, the value set in the decimal_in variable is 123.
The input and output properties are also linked. The integer value sent to input is sent back to the service as output.
The cmd and log properties are strings that are linked. When the demo receives a string value for cmd, the value is reflected back as log. The maximum length of these strings is 1024 bytes of UTF-8 after JSON encoding. This means that some characters are counted as multiple bytes.
The oem_host_version property is a string that gives the template version. The version property is a from-device string that provides the version of code running on the device.
node_batch_hold is a boolean property used to demonstrate batch datapoints. When set to 1, a batch is created and further changes to the properties "log", "output", and "decimal_out" are batched until the batch space is full (a compiled-in limit) or until node_batch_hold is set back to 0.
The stream_up property is a from-device file property that the device sends when the stream_up_len property is non-zero.
The stream_up_len property is an integer property used to demonstrate file properties. When set to a non-zero value, a test pattern of the length given by stream_up_len is sent on the stream_up property, and then stream_up_len is set back to 0 by the device.
The stream_down property is a to-device file property used to demonstrate how to receive a file property. When a new value is received, the demo compares the value with a test pattern and then sets the properties stream_down_len and stream_down_match_len accordingly.
The stream_down_match_len property is a from-device integer property that gives the initial number of bytes matching a test pattern for the stream_down property. It is set by the device whenever a new value is received for the stream_down property.
The stream_down_len property is a from-device integer property that gives the length of the last value received on the stream_down property.
sched1 through sched5 are the five schedules associated with the demo. These schedules allow changes to the properties per specific time settings. For example, you could program the Blue_LED property to turn on every 10 seconds, or once a day, and so on.

Message properties

The ledevb demo supports message properties, which can send very long messages of types: string, JSON, or binary. The properties for this feature are:

message_start: This is a string-type normal (not message) to-device string property used as a command string to start a post of a message property from the device.
Setting it to a string like "1000 json_out" will cause a pattern of 1000 bytes to be sent on the json_out property. The module will set message_start back to an empty string to avoid repeating the command on the next reboot.
binary_in, to-device: Accepts binary data and if it is not longer than 4096 bytes (adjustable in the source), it will send the value back on binary_out. In any case it prints a CRC on the console.
binary_out, from-device: Sends binary data as started with message_start or binary_in.
string_in, to-device: Accepts UTF-8 string data and if it is not longer than 4096 bytes (adjustable in the source), it will send the value back on string_out. In any case it prints a CRC on the console.
string_out, from-device: Sends string data as started with message_start or string_in.
json_in, to-device: Accepts JSON-formatted data and if it is not longer than 4096 bytes (adjustable in the source), it will send the value back on json_out. In any case it prints a CRC on the console.
json_out, from-device: Sends string data as started with message_start or json_in.

Datapoint Metadata

The ledevb demo illustrates how to send metadata with datapoints by using demo_send_prop_with_meta() when sending the "output" or "log" properties, and by using demo_file_start_send() to for the "stream_up" file property.

The metadata sent for these properties includes the time, two keys, and two values. Please note, these are just examples.

On receiving metadata for the "input", "decimal_in", or "stream_up_len" properties, the keys and values are printed on the console.

Datapoint Acknowledgements

The ledevb demo illustrates how to acknowledge a property by calling ada_sprop_send_ack() when receiving "input", "decimal_in", or "stream_up_len" properties. A count of acks is sent as the message. This is only called if the cloud has requested an ack as indicated by the sprop->ack_id field being non-NULL.

On receiving a message property datapoint, the demo sends an ack with the length in the message.

Note that the template must enable the property to request acks from the device, which is not the default, and in the above example template, ack_enabled is only set for "input" and message properties.

Sample Application Source Files

The sample application source is in ayla_demo/main and is composed of the following C files:

# ls *.c
command.c  demo_bt.c	  demo_ledevb.c  demo_wifi.c  sched_conf.c
conf.c	   demo_common.c  demo_stub.c	 ota.c

demo_ledevb.c

The demo_ledevb.c file contains the code specific to the demo and its properties. This file works as follows:

The demo starts in app_main(), which calls demo_start() which is part of demo_common.c. This initializes the demo and ADA and starts the esp wifi framework and launches an application thread which calls demo_init() and runs demo_idle() in this file.
demo_init() registers event handlers for Wi-Fi and client events. It then uses the API ada_sprop_mgr_init() to register the properties handled by the device through the simple property manager. The table used is called demo_props.
demo_led_set() is used for the setting of the LED properties.
demo_int_set() is used for the setting of the input, decimal_in, and stream_up_len properties.
demo_cmd_set() is used for the setting of the cmd property.
demo_idle() first reads the two buttons, and then does the following:
- Sends the version and oem_host_version properties.
- Enters an idle loop that sleeps for 100 ms.
- Updates the Link LED.
- Samples the buttons.
demo_ledevb_wifi_event_handler() is used to turn off LINK LED when Wi-Fi is disconnected.
demo_client_event() is used to update LINK LED and to log the client connection and disconnection events.

demo_common.c

The demo_common.c file contains the code that is common among most demos running under the ESP32. This file works as follows:

The first call to this file is from main() calling demo_start(), which calls module_init(), prints the software version, and then calls app_framework_start() to trigger callbacks to event_handler().
modules_init() initializes the serial port, the CLI.
event_handler() logs each event as it occurs and handles the WLAN_INIT_DONE and NORMAL_CONNECTED events. When WLAN init is completed, it calls init_nvs() to enable the non-volatile storage command line interface, and then it calls demo_client_start() to initialize the ADA. On the NORMAL_CONNECTED event, the event_handler() starts the ADA client and creates the demo thread which runs in demo_idle_enter().
demo_client_start() calls initialization routines that are part of the ADA interfaces and in conf.c and sched_conf.c.
clock_init() makes sure the clock is in range so that the SSL(mbedtls) certificates are valid.
client_conf_init() is described in the conf.c section below.
client_task_init() starts the ADA client thread.
ada_sched_enable() turns on the schedule handler inside the ADA.
demo_client_start() calls demo_init() in demo_ledevb.c to register the properties.

demo_bt.c

The demo_bt file contains code to enable BLE Wi-Fi provisioning from Aura or other mobile apps. This is much preferred to using AP-mode provisioning. This file works as follows:

The first call is from maint() calling demo_start(), which calls demo_bt_init(). This registers a callback to demo_bt_adb_event_handler() and calls al_bt_init() passing a callback to demo_bt_register_services_cb().
When BLE is initialized it calls demo_bt_register_services_cb().
demo_bt_register_services_cb() registers an identify callback, registers the Ayla service, the connection service, optionally the local control service, and the Wi-Fi configuration service.
demo_bt_adb_event_handler() handles indications of the start and end of BLE Wi-Fi provisioning.

conf.c

The conf.c file loads the ADA client configuration from the ESP32 NVS and stores values for ADA as needed. This file also provides several platform-specific interfaces required by the ADA. The platform-provided interfaces have the prefix "adap_."

ADA configuration variables have hierarchical names separated by slash characters, for example "id/dev_id." These names have the prefix "ada." for startup configuration items, or "ada.f." for factory-configuration item. For example, the full name for the ESP32 NVS would be "ada.f.id/dev_id."

In the conf.c file, client_conf_init() reads the MAC address which the ADA sends to ADS to confirm that the device is not using the same serial number as another device. client_conf_init() also does the following:

Calls sched_conf_load() to load the saved schedules.
Calls adw_conf_load() to load the Wi-Fi configuration.
Sets the enable and get_all flags in the client_conf structure, as well as the poll_interval, which is used in cases where connectivity to the Ayla Notification Service (ANS) is lost.

command.c

The command.c file contains the code for CLI commands. Most of these commands are wrappers for the ADA command interfaces. Several of these CLI commands should only be used for manufacturing setup and should not be used for production devices.

One command in particular should be understood because it affects security and how the factory configuration is set and how factory reset works. The configuration in flash has two parts, a factory configuration and a startup configuration. There are two modes "user" and "setup" mode. In setup mode, all items are saved to the factory configuration. In "user" mode, items are saved to the startup configuration. Once all factory configuration is complete, setup mode should be disabled using the "setup_mode disable". This should be done before Wi-Fi setup so that a factory reset can remove the Wi-Fi configuration. The "setup_mode enable" command is only allowed if there's a password configured in conf.h, currently a fixed string. This should use another solution in a product, and should not follow this example.

ota.c

The ota.c file contains code to demonstrate the OTA firmware download interfaces. A host OTA can be deployed from the developer’s site, and the code in this file receives, installs, and boots this OTA. After you burn the first OTA, the device always boots from an OTA partition. The device never boots from the factory partition after that; unless, you erase the OTADATA partition, which is instructing the second stage bootloader to boot from the OTA. The partition table used in this example has no factory configuration, just two OTA partitions. Other solutions are possible. Refer to the Espressif documentation for partitioning and OTA details.

sched_conf.c

The sched_conf.c file contains code to support schedules that are optional. These schedules allow actions on properties to be set for a regular or irregular time frame. For example, you could set a schedule to turn on an output property once every 10 seconds or on the third Tuesday of every month. Schedules can trigger events at sunrise or sunset or some time before or after those events.

The function sched_conf_load() sets the number of schedules and the name for each one. The names should match the names in the Ayla template on the ADS, and the names known by the mobile application. This function also loads the saved values of any schedules that have been set.

The platform-support function adap_sched_conf_persist() saves any schedules that have been changed by the ADA. The in-memory value of the schedule is read using ada_sched_get_index() and compared to the saved value obtained by adap_conf_get(). If there is a difference, the in-memory value is written using adap_conf_set().

The function sched_conf_name() fills in the configuration name for the value of a schedule, given its index.

The functions adap_sched_run_time_write() and adap_sched_run_time_read() save and restore the last time that the schedules were processed. These functions should put that time in non-volatile RAM (not flash), if possible, to avoid having to re-process events that have already taken place after rebooting the system. This saves power if non-volatile RAM is retained and may be important if schedules are heavily used.

Be sure to validate that the non-volatile storage is valid when reading the value (for the time of the event), as it can be scrambled if power is not maintained. A CRC and magic number could be used. If the value is not valid, 0 should be returned.

The functions demo_sched_action_show(), demo_sched_cli(), and demo_sched_test() together implement the "sched" CLI, which can be used to test a schedule to see when it would trigger events.

<sched-name> <action-count> [<time>]

NOTE: The schedule name and action-count are required, and the time is optional.

This shows the next N actions for the named schedule, starting from the current time or any specified time.

demo_stub.c

The demo_stub.c file contains sample implementation for interfaces required by the embedded agent from the Wi-Fi code. This is only used if Ayla’s Wi-Fi code (ADW) is not used.

demo_wifi.c

The demo_wifi.c file initializes the ADW configuration for the AP-mode SSID and the (now obsolete) iOS setup app. This file contains an event handler that receives the notifications of significant Wi-Fi events.

Library Code

The example application links with the following Ayla libraries residing in components/ayla:

Library	Notes
libada	Ayla Integrated agent, see below.
libadb	Ayla Bluetooth library
libayla	See Integrated Agent.
libadw	See Ayla Device Wi-Fi for Embedded Systems
libjsmn	JSMN (pronounced Jasmine) is a small JSON parser. ADA uses a different version of JSMN than the one in ESP32. The ADA version is in `components/ayla/jsmn`.
libnet	This library contains a thin layer of macros and functions that interface with LwIP and ESP32.

Library ada

Below are summaries of the files in this library to help you with ESP32 development. For a full description of this library, see Integrated Agent.

`al/esp32/ada_lock.c`

This file implements an interface to the thread mutexes.

`al/esp32/client_task.c`

This file contains code for the client thread and its initialization, timers, locking, and work queue.

The function ada_init() does the following:

Initializes the logging subsystem, including setting severity levels to be logged for each subsystem.
Initializes the timers and the mutex, which protects most of the client data structures.
Calls init functions for various subsystems.
Calls ada_conf_load() to read configuration settings.
Creates the client_task_queue and the client thread.

The client thread runs in the function client_idle(). This function initializes the TLS library, the client_redir_client_html() handler (described below) and the 'client_task_watchdog' function. Then it takes the client lock and enters the idle loop.

The idle loop handles any pending timers that need to be handled in the client thread while still holding the client lock. Then, the idle loop waits for callbacks on the client_task_queue. This client_task_queue waits until the next scheduled timer or until an event occurs. If a callback is found, its handler is called and the loop repeats.

The al/esp32/client_task.c file contains code in the function client_redir_client_html() that redirects HTTP requests for “/client” to “/client.html”. This is necessary for same-LAN web-based device registration.

This file also contains the client_task_watchdog timer, which periodically checks that the client is healthy, and if so, calls the ESP-IDF function esp_task_wdt_reset().

The remainder of the code in the al/esp32/client_task.c file implements the client timers and synchronization needed with the server thread:

The functions client_lock_int() and client_unlock_int() implement the locking around the client structures. These functions keep some debugging information that can be useful if there is trouble in this area.
The global string pointer client_mutex_func holds the name of the function that last successfully locked the client_lock.
The global integer client_mutex_line gives the line number in that function.
The global pointer client_mutex_owner is a pointer to the OS task that holds the mutex and is used to detect recursive attempts on the lock.

`al/esp32/http_client.c`

The al/esp32/http_client.c file implements the interface between the HTTP client of ADA and the httpc functionality of the ESP32.

`al/esp32/log.c`

The al/esp32/log.c file loads the configuration of the logging system in ADA.

`al/esp32/notify_task.c`

The al/esp32/notify_task.c implements timers for the notification subsystem, which interacts with ANS.

`al/esp32/stubs.c`

The al/esp32/stubs.c file contains miscellaneous required routines - some for features that are not functional in this architecture, and some that are simple interfaces between ADA and ESP32.

`al/net/server.c`

The al/net/server.c file implements an HTTP server that binds to TCP port 80. This is used for Wi-Fi setup and LAN mode. The httpc functionality in ESP32 is not usable while this server is running.

Host Application configuration

The host application is responsible for configuring the ESP-IDF build to include the proper subsystems with the correct settings. An example configuration is provided with the demo host application.

For proper operation of the client health check code (see al/esp32/client_task.c), the task watchdog reset feature should be enabled. It should be configured to panic if not reset within 25 seconds. This can be tested with the "crash hang" command found in the demo code.

Visibility:	private
Name:	demo-esp32-msgs 1.10
Description:	Demo ESP32 msgs 1.10
Registration:	AP-Mode
Model:	demo-esp32 or demo-esp32c3
Version:	demo_esp_msg 1.10
Type:	Wifi