Using the SDK in a Firebase function (google cloud function)

[NicolasGorga]

Hello!

A little context:

The place I'm working at needed to migrate it's IoT solution, which was implemented using GCP IoT (it's getting deprecated by Google on August '23). They have a Firebase project, which includes an Angular Web App to send commands and see information about their device fleet and Firebase functions, which act as the API for IoT functionality and other services (user management, stock module and other things unrelated to IoT).

I've successfully managed to replicate almost all the previous GCP IoT functionality with AWS. My problem is specifically with a 'block communication' functionality. Basically there's a button in the Web App, where the user can click and the target device will not be able to communicate with the MQTT broker until it gets 'unblocked'. When the button is clicked, a Firebase function gets triggered which takes care of performing the necessary actions.

With GCP IoT Core SDK, blocking or unblocking a device is pretty straightforward, but this is not the case for AWS. Now, bare with me, I'm almost getting to the point of my problem, but first:

To block the device using AWS IoT, the new Firebase function I created initially changed the certificate attached to the device status to 'INACTIVE'. Although this prevents the device from connecting again until its certificate status is changed to 'INACTIVE', if the device had an active connection to the broker, this remains after blocking the certificate.

I was suggested here to take advantage of the fact that the broker only allows one connection at a time using the same clientId, so by creating a new connection with the same clientId my target device used, this connection would get dropped. So, I created a certificate for my Firebase function to use (although I find it weird to connect this way, since conceptually its like treating the function as a 'device', even though it isn't) and using the device SDK, create a connection with the same clientId as the one my target device used.

Testing the function locally with Firebase emulators this works like a charm, since the broker sees the new connection with the same clientId I see my device gets disconnected as expected. Finally the problem: **After deploying my function to Firebase, when I see the logs of the Google Cloud function, it fails with message: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE, TLS (SSL) negotiation failed **

I corroborated the certificates are reachable by the deployed function and are the same I used when invoking it locally through the emulators. I'm at a loss here and would appreciate any help you can give me, since this is the only thing stopping me to finally migrate the IoT system to AWS.

Thanks for reading this far!

[bretambrose]

Assuming firebase routes standard out to some kind of logging solution (like AWS Lambda does), can you enable logging (https://github.com/awslabs/aws-crt-nodejs/blob/main/lib/native/io.ts#L87) and attach the results for a failed run?

[NicolasGorga]

Hey @bretambrose, thanks for the quick reply!

After enabling logging, I can now see an additional message that happens before the previously described error. Attached you can see in the first image a WARN, about a certificate not being trusted. The certificates are the same as the ones running locally.

Do you know what could that mean?

[bretambrose]

Given that Android (via Google) didn't/doesn't include AmazonRootCA1.pem in its system trust store for the longest time (as far as I know, it still might not be there, haven't checked in 5+ years), there's a good chance that firebase might not have a system trust store that includes Amazon's root CA. If that's the case and if you're not packaging AmazonRootCA1.pem into your function's data set and doing a manual override of the CA at creation time, that would explain why it works locally (where the system trust store has AmazonRootCA1) but not remotely.

[NicolasGorga]

I'm pretty new to all of this (been using cloud fucntions for 1 week) so bare with me. I'm uploading the device certificate, private key and the amazon root certificate to a folder that is accesible by my function (is that the 'function data set'?).

How would yo do the 'manual override of the CA at creation time'?

Thanks again for your help!

[bretambrose]

That's the idea. Can you paste the whole code block where you're configuring the connection?

[NicolasGorga]

Yes, here it is. Sorry I tried editing the comment but can't get the format right for the code block:

This codes starts the connection process, whenever a user clicks a 'blockDevice' button in the Angular web app, that triggers the Cloud Function (read below to see definition of static method makeIoTDeviceConnection):
`
if (block) {
const blockedCommunication = await IoTService.makeIoTDeviceConnection({
clientId: deviceId,
endpoint: AWS_MQTT_ENDPOINT,
connectionPackage: {
folderPath: join(__dirname, 'iot', 'web-app-connection-package'),
credentialsFileNames: {
publicCertificate: 'certificate.pem.crt',
privateKey: 'private.pem.key',
caCertificate: 'AmazonRootCA1.pem',
},
},
});

  if (!blockedCommunication) {
    throw(new Error(`Failed to block existent connection for device: ${deviceId}`))
  }

  console.log(`Closed existent connection for device: ${deviceId}`);

}
`

Function that makes the MqttConnectionConfig object (in the three paths is where the device certificate, the private key and the amazonrootca1 files reside):

`private static createIoTDeviceConfig(deviceClient: IoTAwsDeviceClient) : mqtt.MqttConnectionConfig {
const { connectionPackage } = deviceClient;
const { credentialsFileNames } = connectionPackage;
const { folderPath } = connectionPackage;

const publicCertificatePath = join(folderPath, credentialsFileNames.publicCertificate);
const privateKeyPath = join(folderPath, credentialsFileNames.privateKey);
const caCertificatePath = join(folderPath, credentialsFileNames.caCertificate);

const config_builder = iotDevice.AwsIotMqttConnectionConfigBuilder.new_mtls_builder_from_path(publicCertificatePath, privateKeyPath);
config_builder.with_endpoint(deviceClient.endpoint);
config_builder.with_port(8883);
config_builder.with_certificate_authority_from_path(caCertificatePath);
config_builder.with_client_id(deviceClient.clientId);

const config = config_builder.build();
return config;

}`

Function that establishes the connection:

`
private static async makeIoTDeviceConnection(deviceClient: IoTAwsDeviceClient) {
try {
io.enable_logging(io.LogLevel.DEBUG)
const config = IoTService.createIoTDeviceConfig(deviceClient);
const client = new mqtt.MqttClient(new io.ClientBootstrap());
const connection = client.new_connection(config);

  connection.on('connect', async () => {
    console.log(`Device ${config.client_id} disconnected!`);
    //The target device should be disconnected by now, disconnect the Web App from the broker
    await connection.disconnect();
  });

  return await connection.connect();

} catch (err) {
  console.error(err);
  throw(err);
}

}
`

[bretambrose]

Quick skim, don't see you doing anything wrong. When you make a local run that is successful, what platform are you on (mac/linux/windows)? If you're not doing it from linux, can you try a "local" run from linux? The TLS paths/implementations are different per platform.

[NicolasGorga]

@bretambrose When doing a local run with the Emulators I'm using Windows. Do Cloud Functions run on Linux? For this local run I would install WSL and try running it there?

[NicolasGorga]

After installing WSL and node js inside the default Ubuntu distribution I downloaded the WSL extension of VS Code and run my Firebase Functions project through Linux, the Functions emulator fails to start. Attached is the image of the message it shows. I've searched around but haven't found a way to solve this so I can continue with your indications, any ideas?

[bretambrose]

Firebase cloud functions will get executed on Linux, or at least yours is. On Linux, s2n will be used for TLS, while on Windows, the built-in SecureChannel API will get used.

I don't really have any insight with WSL or emulator details.

Returning to the original error, the line in question is here: https://github.com/aws/s2n-tls/blob/v1.3.37/tls/s2n_x509_validator.c#L529

Based on the context, s2n is failing the TLS handshake because it doesn't trust the certificate chain sent by the server. There's not much more information as it stands since it's the default arm of the switch statement, which itself is just checking the return value of a call into OpenSSL/libcrypto (X509_verify_cert).

If you can repro the failure on Linux, we might be able to dig further into this.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.