HTTPS GET request - ESP8266 - ENC28j60

[OnJuanTrack]

Hi, I'm traying to compile the SSLClient library in my ESP8266 but I have this message:

Several libraries were found for "EthernetLarge.h"
using: /home/xx/Arduino/libraries/EthernetLarge-master
Several libraries were found for "SSLClient.h"
using: /home/xx/Arduino/libraries/SSLClient
Several libraries were found for "SPI.h"
using: /home/xx/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/SPI
exit status 1

Error compiling for card NodeMCU 1.0 (ESP-12E Module).

I'm wanna use the SSLClient library to query a HTTPS web service using an ESP8266 (NodeMCU 1.0 ESP-12E) with a ENC28j60 (Ethernet module), I wanna know if this is possible.

Thanks.

[prototypicalpro]

Hello!

The EthernetLarge library doesn't compile on the ESP8266, and more importantly doesn't support the ENC8j60 module. You'll want to replace EthernetLarge with a library that supports your module, such as UIPEthernet. I went ahead and modified the EthernetHTTPS example sketch to work with UIPEthernet:

/*
  Web client

 This sketch connects to a website (http://www.arduino.cc/asciilogo.txt)
 using an Arduino Wiznet Ethernet shield.

 Circuit:
 * Ethernet shield attached to pins 10, 11, 12, 13

 created 18 Dec 2009
 by David A. Mellis
 modified 9 Apr 2012
 by Noah Koontz, based on work by Adrian McEwen and Tom Igoe

 */

  // NOTE: This example REQUIRES the EthernetLarge library.
  // You can get it here: https://github.com/OPEnSLab-OSU/EthernetLarge

#include <SPI.h>
#include <UIPEthernet.h>
#include <SSLClient.h>
#include "trust_anchors.h"

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(54,85,55,79);  // numeric IP for Google (no DNS)
const char server[] = "www.arduino.cc";    // name address for Arduino (using DNS)
const char server_host[] = "www.arduino.cc"; // leave this alone, change only above two

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192, 168, 0, 177);
IPAddress myDns(8, 8, 8, 8);

// Choose the analog pin to get semi-random data from for SSL
// Pick a pin that's not connected or attached to a randomish voltage source
const int rand_pin = 10;

// Initialize the SSL client library
// We input an EthernetClient, our trust anchors, and the analog pin
EthernetClient base_client;
SSLClient client(base_client, TAs, (size_t)TAs_NUM, rand_pin);
// Variables to measure the speed
unsigned long beginMicros, endMicros;
unsigned long byteCount = 0;
bool printWebData = true;  // set to false for better speed measurement

void setup() {
  // You can use Ethernet.init(pin) to configure the CS pin
  Ethernet.init(10);  // Most Arduino shields
  //Ethernet.init(5);   // MKR ETH shield
  //Ethernet.init(0);   // Teensy 2.0
  //Ethernet.init(20);  // Teensy++ 2.0
  //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
  //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet

  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  
  // start the Ethernet connection:
  Serial.println("Initialize Ethernet with DHCP:");
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
  } else {
    Serial.print("  DHCP assigned IP ");
    Serial.println(Ethernet.localIP());
  }
  // give the Ethernet shield a second to initialize:
  delay(2000);
  
  Serial.print("connecting to ");
  Serial.print(server);
  Serial.println("...");

  // if you get a connection, report back via serial:
  auto start = millis();
  // specify the server and port, 443 is the standard port for HTTPS
  if (client.connect(server, 443)) {
    auto time = millis() - start;
    Serial.print("Took: ");
    Serial.println(time);
    // Make a HTTP request:
    client.println("GET /asciilogo.txt HTTP/1.1");
    client.println("User-Agent: SSLClientOverEthernet");
    client.print("Host: ");
    client.println(server_host);
    client.println("Connection: close");
    client.println();
  } else {
    // if you didn't get a connection to the server:
    Serial.println("connection failed");
  }
  beginMicros = micros();
}

void loop() {
  // if there are incoming bytes available
  // from the server, read them and print them:
  int len = client.available();
  if (len > 0) {
    byte buffer[80];
    if (len > 80) len = 80;
    client.read(buffer, len);
    if (printWebData) {
      Serial.write(buffer, len); // show in the serial monitor (slows some boards)
    }
    byteCount = byteCount + len;
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    endMicros = micros();
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
    Serial.print("Received ");
    Serial.print(byteCount);
    Serial.print(" bytes in ");
    float seconds = (float)(endMicros - beginMicros) / 1000000.0;
    Serial.print(seconds, 4);
    float rate = (float)byteCount / seconds / 1000.0;
    Serial.print(", rate = ");
    Serial.print(rate);
    Serial.print(" kbytes/second");
    Serial.println();

    // do nothing forevermore:
    while (true) {
      delay(1);
    }
  }
}

In order to run this sketch you'll need to update SSLClient to v1.4.4 and install the UIPEthernet library. I don't have the hardware to test the sketch, but I was able to compile it without issues. Let me know if it works!

On another note, it looks TLS over ethernet is being integrated into the ESP8266 core: esp8266/Arduino#6680, so in the future you won't need SSLClient. Oh well.

[OnJuanTrack]

Hi prototypicalpro, thanks for your fast response, now I have the 1.4.4 version (Downloaded from https://www.arduinolibraries.info/libraries/ssl-client). With the new library <UIPEthernet.h> it works ok, I can connect with my LAN network and I have an IP but when I try connect with the server "if (client.connect("jsonplaceholder.typicode.com", 443)) {" using SSL conection I receive an error in the Serial Monitor, I reduce the code to understand it better, my code is below:

NOTE: when I compile this code to my NodeMCU 1.0 (ESP-12E Module) I receive this message: ATTENTION: The SSLClient-1.4.4 library is intended to run on samd architecture (s) and may be incompatible with your current card which runs on esp8266 architecture (s).

`#include <SPI.h>
#include <UIPEthernet.h>
#include <SSLClient.h>
#include "trust_anchors.h"

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
const int rand_pin = 10;

EthernetClient base_client;
SSLClient client(base_client, TAs, (size_t)TAs_NUM, rand_pin);

void setup() {
Serial.begin(115200);
Serial.println();
Serial.println("Started");
Ethernet.begin(mac);
Serial.print("Conected, the IP address is: ");
Serial.println(Ethernet.localIP());
Serial.println("connecting to server...");

if (client.connect("jsonplaceholder.typicode.com", 443)) {
Serial.println("Conected");
} else {
Serial.println("connection failed");
}

}

void loop() {
}`

The Output in my Serial Monitor is:

Started
Conected, the IP address is: 192.168.1.106
connecting to server...
(SSLClient)(SSL_ERROR)(connect): Failed to connect using m_client. Are you connected to the internet?
connection failed

When I change the line to connect with the server by this, I can connect with the page:

"if (base_client.connect("jsonplaceholder.typicode.com", 80)) {"

Then my ENC28j60 is working ok with the ESP8266 and I can query a web service using HTTP.

//------------------------------------------------------------------------------------------------------------------------//
I also use your code, and this is what I receive in my Serial Monitor:

DHCP assigned IP 192.168.1.104
connecting to www.arduino.cc...
(SSLClient)(SSL_ERROR)(connect): Failed to connect using m_client. Are you connected to the internet?
connection failed
(SSLClient)(SSL_ERROR)(available): Cannot operate if the write error is not reset:
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL
(SSLClient)(SSL_ERROR)(connected): Not connected because write error is set
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL

disconnecting.

ets Jan 8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x4010f000, len 1392, room 16
tail 0
chksum 0xd0
csum 0xd0
v3d128e5c
~ld
Initialize Ethernet with DHCP:
DHCP assigned IP 192.168.1.104
connecting to www.arduino.cc...
(SSLClient)(SSL_ERROR)(connect): Failed to connect using m_client. Are you connected to the internet?
connection failed
(SSLClient)(SSL_ERROR)(available): Cannot operate if the write error is not reset:
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL
(SSLClient)(SSL_ERROR)(connected): Not connected because write error is set
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL

disconnecting.

//---------------------------------------------------------------------------------------//

I dont know why my microcontroller is restarting with your example.

Thanks for your help friend.

[prototypicalpro]

That's interesting. The code where the SSLClient is failing is here (SSLClient.cpp:21):

...
const bool connect_ok = get_arduino_client().connect(host, port);
if (!connect_ok) {
    m_error("Failed to connect using m_client. Are you connected to the internet?", func_name);
    setWriteError(SSL_CLIENT_CONNECT_FAIL);
    return 0;
}
...

All get_arduino_client() does is get a reference to base_client, so this code should be equivalent to:

...
if(!base_client.connect(host, port))
...

However this would suggest that your test base_client.connect("jsonplaceholder.typicode.com", 80) should have failed as well. Given that it succeed, I'm a bit stumped on what the problem could be.

Could you try running base_client.connect("jsonplaceholder.typicode.com", 443), just to see if it connects? If that doesn't work, I'll do some more investigation,

[OnJuanTrack]

Hi my friend,
I've tried your suggest base_client.connect("jsonplaceholder.typicode.com", 443), when I put this code, I receive a "Connected" response, but when I send my GET request (the code is below) I receive a 400 bad request:

// ---------- Code:

void setup() {
Serial.begin(115200);
Serial.println();
Serial.println("Started");
Ethernet.begin(mac);
Serial.print("Conected, the IP address is: ");
Serial.println(Ethernet.localIP());
Serial.println("connecting to server...");

if (base_client.connect("jsonplaceholder.typicode.com", 443)) {
Serial.println("Connected");

base_client.println("GET /comments?postId=7 HTTP/1.1");
base_client.println("User-Agent: SSLClientOverEthernet");
base_client.println("Host: jsonplaceholder.typicode.com");
base_client.println("Connection: close");
base_client.println();

} else {
Serial.println("connection failed");
}
}

// ---------- Response in serial port:
Started
Conected, the IP address is: 192.168.1.109
connecting to server...
Conected
HTTP/1.1 400 Bad Request
Server: cloudflare
Date: Mon, 30 Dec 2019 14:05:25 GMT
Content-Type: text/html
Content-Length: 253
Connection: close
CF-RAY: -

<title>400 The plain HTTP request was sent to HTTPS port</title>

400 Bad Request

The plain HTTP request was sent to HTTPS port

---

cloudflare

This is normal because the UIPEthernet send this information (Get request) in HTTP way, so this request can't arrive in the 443 port (https port).

Thanks for your help my friend.

[prototypicalpro]

Alright, so we know for sure that UIPEthernet is not the problem. I made some changes and released v1.4.6, could you update SSLClient and give the sketch I sent you another shot?

[OnJuanTrack]

Hi friend,

I've uploaded your code, only I've commented this line Ethernet.init(10); all the code is equal. This is what I'm receiving in my serial port:

//-----------------------------------------------------------------------

Initialize Ethernet with DHCP:
DHCP assigned IP 192.168.1.109
connecting to www.arduino.cc...

User exception (panic/abort/assert)
Panic core_esp8266_main.cpp:121 __yield

stack>>>

ctx: sys
sp: 3ffffce0 end: 3fffffb0 offset: 0000
3ffffce0: 00040000 00000000 000043dd 00000030
3ffffcf0: 000000fe 00000000 00000000 00000000
3ffffd00: 00000000 00000000 00000000 3fff4e28
3ffffd10: 00000800 3fff30a0 00000001 0000001f
3ffffd20: 000000a0 3fff4a0c 00000003 402129f2
3ffffd30: 007a1200 25043793 00000000 40212a53
3ffffd40: 007a1200 00000001 0000000f 40212525
3ffffd50: 000000a0 3fff4a0c 00000003 4020f630
3ffffd60: 00000080 3fff4a0c 00000008 00000004
3ffffd70: 3fff4968 3ffe84d3 00000020 4020f733
3ffffd80: 3fff28a0 3fff2868 00000039 4020f75d
3ffffd90: 00000200 00000093 3fff4ad7 4020f826
3ffffda0: 3ffe84d4 3fff4a0c 3fff43dd 3fff3084
3ffffdb0: 00000000 3fff32a0 3fff28a0 00000004
3ffffdc0: 3fff4968 3fff2868 3fff4ad7 4020ec7e
3ffffdd0: 4024ace5 3fff2868 3fff49e0 4024ace5
3ffffde0: 4024ace5 3fff2868 3fff28a0 00000004
3ffffdf0: 3fff4968 3fff2868 3fff49e0 4020e4d4
3ffffe00: 00000000 3fff2868 3fff49e0 4020e68f
3ffffe10: 4024ace5 3fff2868 00000000 40210b79
3ffffe20: 000004e7 00000000 4bc6a7f0 00000000
3ffffe30: 3fff28a0 3fff41dd 00003e20 00000200
3ffffe40: 00000000 00000004 3fff2868 00000004
3ffffe50: 00000000 00000004 3fff2868 40210d33
3ffffe60: 000007fb 00000005 3fff28a0 3fff2da0
3ffffe70: 000007fb 00000008 00001c92 00000000
3ffffe80: 00000001 3ffe8746 3fff28a0 40202dbe
3ffffe90: 3fffff14 00000010 4010026a 3fff2868
3ffffea0: 4024ad01 00000000 3fff28a0 4021123a
3ffffeb0: 3fff4ad0 3fff5318 00001c89 4020ed40
3ffffec0: 00000000 3fffff10 00000001 4024ad38
3ffffed0: 000001bb 3fff49e0 3fff49e0 4024ad38
3ffffee0: 000001bb 3fff49e0 3fff52b8 4020e55a
3ffffef0: 2fbf1468 0000002e 3fff5fb4 4024ad38
3fffff00: 4020e4f4 000001bb 3fff49e0 4020e61c
3fffff10: 40214328 00000000 00000000 00000000
3fffff20: 00000000 000003e8 3ffe8400 00000000
3fffff30: 3ffe8746 4bc60000 40214328 4024ad38
3fffff40: 3ffe8746 00000001 3fff2868 4024ad38
3fffff50: 3ffe8746 00000001 3fff2868 40211436
3fffff60: 00000000 3fff4a60 3fff4cb8 00001969
3fffff70: 3ffe8746 3fff2868 3fff4cb8 4020d300
3fffff80: 40214328 6d01a8c0 feefeffe feefeffe
3fffff90: feefeffe feefeffe feefeffe 3fff4de8
3fffffa0: 3fffdad0 00000000 3fff4da8 402125f4
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1392, room 16
tail 0
chksum 0xd0
csum 0xd0
v3d128e5c
~ld

Initialize Ethernet with DHCP:
DHCP assigned IP 192.168.1.109
connecting to www.arduino.cc...
(SSLClient)(SSL_ERROR)(connect): Failed to connect using m_client. Are you connected to the internet?
connection failed
(SSLClient)(SSL_ERROR)(available): Cannot operate if the write error is not reset:
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL
(SSLClient)(SSL_ERROR)(connected): Not connected because write error is set
(SSLClient)(SSL_ERROR)(m_print_ssl_error): SSL_CLIENT_CONNECT_FAIL

disconnecting.

[prototypicalpro]

Alright, so Panic core_esp8266_main.cpp:121 __yield leads to this file, which after some investigation seems to be checking for two things: Is this function being called in an ISR, and has the stack been corrupted. Since neither of of us are using ISRs in our code, my best guess is that SSLClient is attempting to use too much memory and overflowing the stack. This isn't too surprising, as SSLClient was designed to put as little memory into the heap as possible.

Unfortunately, since this was a design choice, reducing the memory usage of SSLClient is not an easy fix. I can do some additional investigation of this issue, however your best bet is probably going to be to wait for this pull request to the ESP8266 core which will add support for SSL to all network interfaces. I will add a note to the README that the ESP8266 is not supported until I can address this issue.

Sorry I couldn't be of more help. Send me an update if you make any progress.

[MeY0u]

hello, i am facing the same problem after 4 years.
i have tried EthernetENC and UIPEthernet libraries with the example above with ENC28j60 and both return
(SSLClient)(SSL_ERROR)(m_update_engine): Error writing to m_client

no idea how to improve it. any ideas?