Giter VIP home page Giter VIP logo

bojanjurca / multitasking-esp32-http-ftp-telnet-servers-for-arduino Goto Github PK

View Code? Open in Web Editor NEW
89.0 11.0 18.0 3.2 MB

ESP32 with HTTP server, Telnet server, file system, FTP server FTP client, SMTP client, cron daemon and user management

License: Creative Commons Zero v1.0 Universal

C++ 68.64% C 22.27% HTML 8.71% Batchfile 0.01% VBScript 0.39%
esp32-arduino ftp-server telnet-server web-server real-time-clock oscilloscope user-management websockets tcp-server tcp-client

multitasking-esp32-http-ftp-telnet-servers-for-arduino's Introduction

ESP32 with HTTP server, Telnet server, file system, FTP server FTP client, SMTP client, cron daemon and user management.

This template is a quick and easy way to build a nice user interface for an ESP32 project, without having to take care of all the switches, LED diodes, displays, etc.

  • You only have to modify the telnetCommandHandlerCallback function to build the Telnet user interface for your project.

  • You only have to modify the httpRequestHandler function to build the WEB user interface for your project. If you want a nice stylish look and feel there is also a FTP server built into the template so you can upload larger HTML files.

  • Dmesg message logging is built-in which can help you see what is going on at run-time and debug your project (accessible through Telnet server).

  • A Web-based Oscilloscope is already built-in which can help you see the signals on your Esp32 pins at run-time (accessible through Http server).

Demo ESP32 server is available at http://jurca.dyn.ts.si

The latest changes

The latest changes are mainly about porting the code to IDF 5.x. If you are looking for code that runs on IDF 4.x please, download the release v2.04.

There are still some parts that I was unable to port to IDF 5.x well enough so far. For example obtaining IP addresses of stations connected to the AP interface in iw Telnet command. Any help with this would be appreciated.

Screenshot

Fully multitasking HTTP server

HTTP server can handle HTTP requests in two different ways. As a programmed response to (for example REST) requests or by sending .html files from /var/www/html directory. Cookies and WebSockets are also supported to certain extent. Demo HTTP server is available at 193.77.159.208.

HTTP server performance

HTTP server performance

Fully multitasking Telnet server

Telnet server can, similarly to HTTP server, handle commands in two different ways. As a programmed response to some commands or it can execute already built-in commands (like ls, ping, ...). There is also a very basic text-editor built in, mainly for editing small configuration files. Demo Telnet server is available at 193.77.159.208 (login as root with default password rootpassword).

Fully multitasking FTP server

FTP server is needed for uploading configuration files, .html files, ... to ESP32 file system. Both active and passive modes are supported.

Time zones

All the time zones form https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv can be used.

Simple key-value database

A simple key-value database is included for the purpose of managing (keeping) web session tokens. Other key-value databases can easily be added using the same technology, for different purposes (such as statistics of visited pages, etc).

Configuration files

/usr/share/zoneinfo                       - contains (POSIX) timezone information
/etc/passwd                               - contains users' accounts information
/etc/shadow                               - contains hashed users' passwords
/network/interfaces                       - contains WiFi STA(tion) configuration
/etc/wpa_supplicant/wpa_supplicant.conf   - contains WiFi STA(tion) credentials
/etc/dhcpcd.conf                          - contains WiFi A(ccess) P(oint) configuration
/etc/hostapd/hostapd.conf                 - contains WiFi A(ccess) P(oint) credentials
/etc/ntp.conf                             - contains NTP time servers names
/etc/crontab                              - contains scheduled tasks
/etc/mail/sendmail.cf                     - contains sendMail default settings
/etc/ftp/ftpclient.cf                     - contains ftpPut and ftpGet default settings

Setup instructions

  1. Copy all files in this package into Esp32_web_ftp_telnet_server_template directory.
  2. Open Esp32_web_ftp_telnet_server_template.ino with Arduino IDE.
  3. modify (some or all) the default #definitions in Esp32_servers_config.h file (that will be later written to configuration files) before the sketch is run for the first time:
// include version_of_servers.h to include version information
#include "./servers/version_of_servers.h"
// include dmesg_functions.h which is useful for run-time debugging - for dmesg telnet command
#include "./servers/dmesg_functions.h"


// 1. TIME:    #define which time settings, wil be used with time_functions.h - will be included later
    // define which 3 NTP servers will be called to get current GMT (time) from
    // this information will be written into /etc/ntp.conf file if file_system.h will be included
    #define DEFAULT_NTP_SERVER_1                      "1.si.pool.ntp.org"   // <- replace with your information
    #define DEFAULT_NTP_SERVER_2                      "2.si.pool.ntp.org"   // <- replace with your information
    #define DEFAULT_NTP_SERVER_3                      "3.si.pool.ntp.org"   // <- replace with your information
    // define time zone to calculate local time from GMT
    #define TZ                                        "CET-1CEST,M3.5.0,M10.5.0/3" // or select another (POSIX) time zones: https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv


// 2. FILE SYSTEM:     #define which file system you want to use 
    // the file system must correspond to Tools | Partition scheme setting: FAT for FAT partition scheme, LittleFS for SPIFFS partition scheme)
    #define FILE_SYSTEM    FILE_SYSTEM_LITTLEFS // or FILE_SYSTEM_FAT


// 3. NETWORK:     #define how ESP32 will use the network
    // STA(tion)
    // #define how ESP32 will connecto to WiFi router
    // this information will be written into /etc/wpa_supplicant/wpa_supplicant.conf file if file_system.h will be included
    // if these #definitions are missing STAtion will not be set up
    #define DEFAULT_STA_SSID                          "YOUR_STA_SSID"       // <- replace with your information
    #define DEFAULT_STA_PASSWORD                      "YOUR_STA_PASSWORD"   // <- replace with your information
    // the use of DHCP or static IP address wil be set in /network/interfaces if file_system.h will be included, the following is information needed for static IP configuration
    // if these #definitions are missing DHCP will be assumed
    // #define DEFAULT_STA_IP                            "10.18.1.200"      // <- replace with your information
    // #define DEFAULT_STA_SUBNET_MASK                   "255.255.255.0"    // <- replace with your information
    // #define DEFAULT_STA_GATEWAY                       "10.18.1.1"        // <- replace with your information
    // #define DEFAULT_STA_DNS_1                         "193.189.160.13"   // <- replace with your information
    // #define DEFAULT_STA_DNS_2                         "193.189.160.23"   // <- replace with your information

    // A(ccess) P(oint)
    // #define how ESP32 will set up its access point
    // this information will be writte into /etc/dhcpcd.conf and /etc/hostapd/hostapd.conf files if file_system.h will be included
    // if these #definitions are missing Access Point will not be set up
    // #define DEFAULT_AP_SSID                           HOSTNAME           // <- replace with your information,
    // #define DEFAULT_AP_PASSWORD                       "YOUR_AP_PASSWORD" // <- replace with your information, at least 8 characters
    // #define DEFAULT_AP_IP                             "192.168.0.1"      // <- replace with your information
    // #define DEFAULT_AP_SUBNET_MASK                    "255.255.255.0"    // <- replace with your information

    // define the name Esp32 will use as its host name 
    #define HOSTNAME                                   "MyEsp32Server"      // <- replace with your information,  max 32 bytes
    #define MACHINETYPE                                "ESP32 Dev Module"   // <- replace with your information, machine type, it is only used in uname telnet command


// 4. USERS:     #define what kind of user management you want before #including user_management.h
    #define USER_MANAGEMENT                           UNIX_LIKE_USER_MANAGEMENT // or HARDCODED_USER_MANAGEMENT or NO_USER_MANAGEMENT
    // if UNIX_LIKE_USER_MANAGEMENT is selected you must also include file_system.h to be able to use /etc/passwd and /etc/shadow files
    #define DEFAULT_ROOT_PASSWORD                     "rootpassword"        // <- replace with your information if UNIX_LIKE_USER_MANAGEMENT or HARDCODED_USER_MANAGEMENT are used
    #define DEFAULT_WEBADMIN_PASSWORD                 "webadminpassword"    // <- replace with your information if UNIX_LIKE_USER_MANAGEMENT is used
    #define DEFAULT_USER_PASSWORD                     "changeimmediatelly"  // <- replace with your information if UNIX_LIKE_USER_MANAGEMENT is used


// #include (or comment-out) the functionalities you want (or don't want) to use
#include "./servers/fileSystem.hpp"
#include "./servers/time_functions.h"               // file_system.h is needed prior to #including time_functions.h if you want to store the default parameters
#include "./servers/network.h"                      // file_system.h is needed prior to #including network.h if you want to store the default parameters
#include "./servers/httpClient.h"
#include "./servers/ftpClient.h"                    // file_system.h is needed prior to #including ftpClient.h if you want to store the default parameters
#include "./servers/smtpClient.h"                   // file_system.h is needed prior to #including smtpClient.h if you want to store the default parameters
#include "./servers/userManagement.hpp"             // file_system.h is needed prior to #including userManagement.hpp in case of UNIX_LIKE_USER_MANAGEMENT
#include "./servers/telnetServer.hpp"               // needs almost all the above files for whole functionality, but can also work without them
#include "./servers/ftpServer.hpp"                  // file_system.h is also necessary to use ftpServer.h
#include "./servers/httpServer.hpp"                 // file_system.h is needed prior to #including httpServer.h if you want server also to serve .html and other files
  1. Select one of SPIFFS partition schemas (Tool | Partition Scheme).

  2. Compile the sketch and run it on your ESP32. Doing this the following will happen:

    • ESP32 flash memory will be formatted with the LittleFs file system. FAT file system is supported as well. It is faster but uses (approximately 40 KB) more memory. WARNING: every information you have stored into ESP32's flash memory will be lost.
    • Configuration files will be created with the default settings.
    • Two users will be created: root with rootpassword and webadmin with webadminpassword.

At this point, you can already test if everything is going on as planned by http, FTP or telnet to your ESP32. Your ESP32 is already working as a server but there are a few minor things yet left to be done.

  1. FTP (demo and example: index.html, ...) files from html directory to ESP32's /var/www/html/ directory.

  2. Delete all the examples and functionalities that don't need and all the references to them in the code. They are included just to make the development easier for you.

Debugging the code

Telnet server provides Unix/Linux like dmesg circular message queue. You can monitor your ESP32 behaviour even when it is not connected to a computer with a USB cable. In your C++ code use dmesg () function to insert important messages about the state of your code into a circular queue. When you want to view it, connect to your ESP32 with Telnet client and type dmesg command.

Screenshot

Debugging the signals

ESP32 oscilloscope is a web-based application included in Esp32_web_ftp_telnet_server_template. It is accessible through a web browser once oscilloscope.html is uploaded to ESP32's /var/www/html directory.

Screenshot

multitasking-esp32-http-ftp-telnet-servers-for-arduino's People

Contributors

bojanjurca avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

multitasking-esp32-http-ftp-telnet-servers-for-arduino's Issues

after a while LED control was rendered malfunction

Testing perhaps 30 minutes or an hour or so, the LED control was no longer functional, unless reboot the ESP32. Any idea for debugging this?. It is latest screen capture of following and remain no control to LED. thanks.
buildin_LED_is_faded

ESP32 stream function is working conditionally with different browser

hi, ESP32 stream function is working conditionally with different browser
Firefox is not working, Win10 Edge ok, Android chrome ok, any idea whether server side issue or the browser variance ?

revised_index

here is screen shot with win10 machine, on the left EDGE is ok, on the right firefox used was ver 86. 32 bit, error?
firefox_NG_Edge_OK

Compile issues

Looks very useful code. I am having issues compiling it. Following are some error messages at the end :

Arduino: 1.8.9 (Windows 7), Board: "LOLIN D32, Default, 80MHz, 921600, None"

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\file_system.h: In function 'bool writeEntireFileWithoutSemaphore(String, char*)':

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\file_system.h:114:3: error: control reaches end of non-void function [-Werror=return-type]

}

^

In file included from C:\Users\WinCycle\Documents\Arduino\Esp32_web_ftp_telnet_server_template\Esp32_web_ftp_telnet_server_template.ino:233:0:

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\telnetserver.hpp: In static member function 'static bool telnetServer::pingRecv(telnetServer::pingDataStructure, TcpConnection, int)':

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\telnetserver.hpp:883:7: error: control reaches end of non-void function [-Werror=return-type]

   }  

   ^

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\telnetserver.hpp: In static member function 'static String telnetServer::curl(TcpConnection*, String, String)':

c:\users\wincycle\appdata\local\temp\arduino_build_857363\sketch\servers\telnetserver.hpp:1064:7: error: control reaches end of non-void function [-Werror=return-type]

   }

   ^

cc1plus.exe: some warnings being treated as errors

Using library WiFi at version 1.0 in folder: C:\Users\WinCycle\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\WiFi
Using library SPIFFS at version 1.0 in folder: C:\Users\WinCycle\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\SPIFFS
Using library FS at version 1.0 in folder: C:\Users\WinCycle\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\FS
exit status 1
format '%d' expects argument of type 'int', but argument 3 has type 'long unsigned int' [-Werror=format=]

pool.ntp.org is not available when FAT used, but SPIFFS was good to see WIFI connected

ntp_error
hi, thanks for this great codepiece.
we have seen the STA connected succefully when SPIFFS is used, however FAT and selected, STA disconnected immedeately and not NTP is capable, any idea to address this issue?

here is the log of FAT and NTP failed,

14:14:04.765 -> [ 86] [file system] mounted.
14:14:04.813 -> [ 143] [cronDaemon] started.
14:14:04.908 -> [ 264] [network] starting WiFi
14:14:04.908 -> [ 265] [network] [STA] connecting STAtion to router using DHCP.
14:14:05.146 -> [ 468] [network] [STA] WiFi client started.
14:14:05.146 -> [ 479] [network] [AP] initializing access point: MyESP32Server/YOUR_AP_PASSWORD, 10.0.1.3, 10.0.1.3, 255.255.255.0
14:14:05.146 -> [ 481] [network] [AP] WiFi access point started.
14:14:05.146 -> E (1014) wifi:sta is connecting, return error
14:14:05.146 -> [ 507] [httpServer] started on 0.0.0.0:80.
14:14:05.193 -> [ 519] [ftpServer] started on 0.0.0.0:21 with firewall.
14:14:05.193 -> [ 531] [telnetServer] started on 0.0.0.0:23.
14:14:05.193 -> [ 532] [loop] free heap: 197620 bytes.
14:14:07.564 -> [ 2927] [network] [STA] disconnected from WiFi.
14:14:10.174 -> [ 5532] [example 08] started.
14:14:19.798 -> [ 15145] 1.si.pool.ntp.org is not available.
14:14:19.798 -> [ 15147] 2.si.pool.ntp.org is not available.
14:14:19.798 -> [ 15149] 3.si.pool.ntp.org is not available.
14:14:20.177 -> [ 15532] [example 08] couldn't get current time from NTP servers (yet)
14:14:20.177 -> [ 15532] [example 09] started.
14:14:20.224 -> [ 15554] [example 09] HTTP/1.1 200 OK

WebSocket binary endianness

Javascript does not comply with TCP rule of sending data over network in network byte order, which is big endian. Most of computers out there are little endian machines just like ESP32 so communication works just fine without considering byte reordering. But as soon as WebSocket client is not little endian machine, binary data exchange will not work anymore without doing some extra work.

Writing a generic solution that works in all cases could be difficult. Personally, I do not have different types of machines at my disposal so I am not in position to do the testing myself. But here are some ideas:

  • Stay at text data exchange. It is insensitive to endianness but less efficient then binary data exchange.
  • Develop application so that exchanges data with server on 1-byte basis. 1 byte is, like text, insensitive to endianness. 1 byte corresponds with int8_t or uint8_t C data types and Int8Array or Uint8Array typed Javascript arrays.
  • If 1 byte information is not enough and you have to use 2 bytes which correspond to with int16_t or uint16_t C data types and Int16Array or Uint16Array typed Javascript arrays, you can use some sort of handshake at the beginning of communication. Javascript could send 1 element Uint16Array with hexadecimal value AABB for example to ESP32 server. The server can check the unsigned integer that arrived.
    • If it is AABB then Javascript client runs on little endian machine like ESP32 itself so no byte reordering is necessary.
    • If it is BBAA then Javascript client runs on big endian machine and ESP32 server should do byte reordering for all incoming and outgoing data:
uint16_t *a = (uint16_t *) buffer;
int n = bytesRead / sizeof (uint16_t);
for (int i = 0; i < n; i ++) a [i] = ntohs (a [i]); // use htons () for outgoing data instead or ntohs () again – it doesn’t really matter
  • In case of elementary data type of 4 bytes which correspond to int32_t or uint32_t or also float C data types and Int32Array or Uint32Array or also Float32Array typed Javascript arrays Javascript could send 1 element Uint32Array with hexadecimal value AABBCCDD for example to ESP32 server. The tings on the ESP32 server side now get even more complicated. If it reads:
    • AABBCCDD then Javascript client runs on little endian machine,
    • DDCCBBAA then Javascript client runs on big endian machine,
    • CCDDAABB then Javascript client runs on mid-big endian machine,
    • BBAADDCC then Javascript client runs on mid-little endian machine.

use SD card instead of flash

The flash is too small,only 4MB.I'm wondering if we can use SD card filesystem,but i can't make it.Do you have any plans to support SD filesystem. :)

warning: ISO C++ forbids converting a string constant to 'char*' [-Wwrite-strings]

hi, seeing compiler warning like this,

warning: ISO C++ forbids converting a string constant to 'char*' [-Wwrite-strings]
cronTabAdd ("0 0 0 1 1 * newYear'sGreetingsToProgrammer"); // triggers at the beginning of each year

cronTabAdd ( "* * * * * * gotTime"); // triggers only once - when ESP32 reads time from NTP servers for the first time cronTabAdd ( "0 0 0 1 1 * newYear'sGreetingsToProgrammer"); // triggers at the beginning of each year cronTabAdd ( "0 * * * * * onMinute"); // triggers each minute at 0 seconds

change to (char *), this will remove those warning,
cronTabAdd ((char *) "* * * * * * gotTime"); // triggers only once - when ESP32 reads time from NTP servers for the first time cronTabAdd ((char *) "0 0 0 1 1 * newYear'sGreetingsToProgrammer"); // triggers at the beginning of each year cronTabAdd ((char *) "0 * * * * * onMinute"); // triggers each minute at 0 seconds

Can't connect to access point (AP) from every client

Some clients fail to connect to ESP32 in AP (access point) mode. For example, Android phone establishes WiFi connection without a problem whereas HP / Windows laptop fails. This issue is already opened: espressif/arduino-esp32#2382. Until it is resolved (hopefully in the next version of Arduino core for ESP32, current version is 1.0.1) a workaround is described there that somewhat fixes the problem.

Compile with ESP-IDF

Can you please guide me to compile this project with ESP-IDF without Arduino component? I am using ESP-IDF v4.4.1.

Some ESP32 boards do not support FAT partition schemes (FAT file system)

Download the latest source files (from repository) where you can choose between FAT and LittleFS file systems. You can run FAT file system only when one of FAT partition schemes is selected (Tools | Partition scheme | ...) and LittleFS only when one of SPIFFS partition schemes is selected (Tools | Partition scheme | ...  ).

Use:

#define FILE_SYSTEM FILE_SYSTEM_LITTLEFS // or (default) FILE_SYSTEM_FAT

prior to

#include "./servers/file_system.h"

Memory problems: memory leak, memory fragmentation, instability at heavy load

There are actually three separate problems identified so far. I'm still working on all of them.

  1. Memory leak is caused by sendBufferSemaphore = xSemaphoreCreateMutex in oscilloscope.h which creates a new semaphore on the heap each time oscilloscope is run. This will be replaced by xSemaphoreCreateBinaryStatic which creates semaphore on the stack.

  2. Memory fragmentation is caused by placing dmesg String messages on the heap. Entire dmesg message queue will be replaced by C strings which reside on the stack.

  3. Instability at heavy load is actually caused by compiler - it crashes ESP32 when there is not enough memory left to create a new object instance or a new task stack. This issue will be bypassed by checking if max free memory block is big enough, before creating a new instance or starting a new task.

W5500 Ethernet Support

Thanks for your great library. I wanted to use ESP32 Ethernet using W5500 chip instead of WiFi, but I had no chance. Even after implementing the W5500 with lwip, the socket function fails and reboots the esp32 again and again. Is there any chance to help me with that?

Error in European DST calculation

Sometimes the time change happens a week too early.

Please correct two #definitions in time_functions.h to look like this:

#define PAST_EUROPE_SPRING_TIME_CHANGE (tStr.tm_mon > 2 /* Apr-Dec */ || (tStr.tm_mon == 2 /* Mar */ && tStr.tm_mday - tStr.tm_wday >= 25 /* last Sun or latter */ && (tStr.tm_wday > 0 /* Mon-Sat */ || (tStr.tm_wday == 0 /* Sun */ && tStr.tm_hour >= 1 /* >= 1:00 GMT */))))

#define PAST_EUROPE_AUTUMN_TIME_CHANGE (tStr.tm_mon > 9 /* Nov-Dec */ || (tStr.tm_mon == 9 /* Oct */ && tStr.tm_mday - tStr.tm_wday >= 25 /* last Sun or latter */ && (tStr.tm_wday > 0 /* Mon-Sat */ || (tStr.tm_wday == 0 /* Sun */ && tStr.tm_hour >= 1 /* >= 1:00 GMT */))))

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.