Smart Home Solution using Face Recognition
An Artificial Intelligence and Android based face recognition app is used to open and close a remote smart door in a smart home. Tensorflow web is used to train the user’s face images and build an AI model. This AI model is converted to mobile friendly — ‘tflite’ format. An Android application is built with this AI model. This app communicates with Blockchain nodes through JSON-RPC and Web3J interface. Two Blockchain miner nodes are running on Google cloud platform based VMs. The third node is a RPI based Blockchain node. These three nodes constitute a P2P network. A smart contract is deployed in this network through which the miner1 node deposits and withdraws tokens to and from the RPI node. A client application which runs on the RPI node monitors these transactions and controls a smart door through the RPI-GPIO port. When the tokens are deposited in the RPI node then the smart door is opened. On the other hand, if the tokens are withdrawn from the RPI node, the smart door is closed. Also, this mini-project involves 7 digital transformation technologies such as AI, Data Science, Blockchain, Android, IoT, Cloud computing and Web3.0.
The following development environments are required:
1. Web application to train face images(HTML/JavaScript)
2. Google Colab to convert web model to TFLite model (Python)
3. Android Studio to build Android application with Web3J (Java)
4. Google cloud VMs (2nos) for running miners (Geth)
5. Raspberry PI as a Blockchain node (Geth)
6. Truflle Environment to build and deploy Smart Contract (Solidity)
7. IOT client application running on RPI (JavaScript/NodeJS)
Pre-requisite
The following medium articles need to be executed as a pre-requisite:
I. Smart Home Solution using Gesture Recognition.
https://anbunathanramaiah.medium.com/smart-home-solution-using-gesture-recognition-3be30071a78f
II. Smart Home Solution using Smart Contract.
https://anbunathanramaiah.medium.com/smart-home-solution-using-smart-contract-7eb7cd3407d3
III. Smart Home Solution using Android and Blockchain. https://anbunathanramaiah.medium.com/smart-home-solution-using-android-and-blockchain-c5d6f23122b0
The steps are given below:
1. Face Recognition
Prepare Face recognition model as per the steps given in section1 — ‘Data Preparation and training’ of the medium article:
2. Capture images
copy the code from:
https://github.com/anbunathan/blockchain_examples/tree/master/face-recognition/web
Open index.html in your Chrome browser.
Set the parameters:
Learning rate = 0.0001, Batch size = 0.4, Epochs = 20 and Hidden Units = 100
Capture images for open, close by clicking their icons
(200 images for each with different lighting conditions)
Open images are captured with the face of user
Close images are captured without the face of user
3. Train the images
Click Train button
Wait for training to complete
Click Test button
Test open, close operations
Download the model by clicking ‘DOWNLOAD MODEL’ button
The following files are downloaded
labels.txt, model.json, model.weights.bin
4. Prepare Tensorflow-Lite model
Convert tensorflowjs model to tflite model based on steps given in section2 — ‘Convert tensorflowjs model to tflite model’ of medium article:
https://anbunathanramaiah.medium.com/smart-home-solution-using-gesture-recognition-3be30071a78f
Copy the code from github for this purpose:
https://github.com/anbunathan/blockchain_examples/tree/master/face-recognition/ml
Once the conversion is done, download model.tflite and labels.txt file from Google colab
5. Android code:
Copy the code from:
https://github.com/anbunathan/blockchain_examples/tree/master/face-recognition/android
In module gradle file, add the folloing
— — — — — — — — — — — — — -
implementation ‘org.web3j:core:4.2.0-android’
— — — — — — — — — — — — —
Add downloaded model.tflite and labels.txt under assets folder:
D:\trial\blockchain_examples\face-recognition\android\app\src\main\assets
In labels.txt,
change open,close to
open
close
I. Changes in the code (ImageClassifier.java)
from
labelProbArray = new float[1][8];
to
labelProbArray = new float[1][2];
II. XML layout changes:
comment out other layout elements such as none
in
res->layout->layout_bottom_sheet.xml
res->layout->layout_direction.xml
III. changes in Camera2BasicFragment.java
comment out other layout elements such as none
// leftLayout = view.findViewById(R.id.none_layout);
// else if (token.contains(“none”)) {
// leftLayout.setEnabled(true);
// leftLayout.setAlpha(1.0f);
// Log.e(“amlan”, “leftLayout enabled “);
//// } else if (token.equalsIgnoreCase(“right”)) {
// }
Change
if(probability<0.8){
finalToken = “NONE”;
}
to
if(probability<0.9){
finalToken = “CLOSE”;
}
// else if (lastSelectedGesture.equalsIgnoreCase(“NONE”)) {
// leftLayout.setBackgroundResource(R.drawable.base);
// }
// else if (token.equalsIgnoreCase(“NONE”)) {
// if (leftLayout.isEnabled()) leftLayout.setBackgroundResource(R.drawable.selection_base);
// }
IV. Comment out code related to ‘leftLayout’
// leftLayout,
// leftLayout.setEnabled(false);
// leftLayout.setAlpha(0.4f);
6. Maintain the state change in Android code:
The state change between ‘open’ and ‘close’ takes 10 seconds for smooth actuator functioning.
— — — — — — — — — — — — — — — — — — — — —
if(stateChanged == false) {
if (token.contains(“open”) && probability >= 0.95) {
if (lastState.contains(“close”)) {
lastState = “open”;
finalToken = “OPEN”;
allowStateChange = false;
stateChanged = true;
try {
String finalToken1 = finalToken;
activity.runOnUiThread(
new Runnable() {
@Override
public void run() {
highLightDirectionButton(finalToken1);
}
});
depositToContract();
}catch(Exception e){
e.printStackTrace();
}
}
}
else {
if (lastState.contains(“open”)) {
lastState = “close”;
finalToken = “CLOSE”;
allowStateChange = false;
stateChanged = true;
try {
String finalToken1 = finalToken;
activity.runOnUiThread(
new Runnable() {
@Override
public void run() {
highLightDirectionButton(finalToken1);
}
});
withdrawFromContract();
}catch(Exception e){
e.printStackTrace();
}
}
if(initialCondition == true){
initialCondition = false;
allowStateChange = true;
lastState = “close”;
finalToken = “CLOSE”;
String finalToken1 = finalToken;
activity.runOnUiThread(
new Runnable() {
@Override
public void run() {
highLightDirectionButton(finalToken1);
}
});
initializeWeb3J();
try {
withdrawFromContract();
}catch(Exception e){
e.printStackTrace();
}
}
}
}
if(stateChanged == true){
try {
new Timer().schedule(new TimerTask() {
@Override
public void run() {
// this code will be executed after 10 seconds
allowStateChange = true;
stateChanged = false;
}
}, 10000);
}catch(Exception e){
e.printStackTrace();
}
}
7. Add Blockchain code in Android app
7.0 add important variables
— — — — — — — — — — — — — — — — — — — — — —
private static String geth_url = “http://35.247.47.68:8042"; //MINER1 IP & PORT
private static String CONTRACT_ADDRESS = “0x0ce678aa4dabef4627fd351deeb35889e6c3d246”; //SMART CONTRACT
private static String PRIVATE_KEY_GETH = “0x57fbf0ad60b682272423afa3c5bd9ed9c459cbf5af3c6f074594ba386ec37428”;
private static String recipient = “0xe6f8bbc22d6ef8cb0991e1a27d9909c92422c119”; //address of RPI
private static Web3j web3j = null;
private static Credentials credentials = Credentials.create(PRIVATE_KEY_GETH);
— — — — — — — — — — — — — — — — — — — — — — — — -
The following functions are added:
7.1 initializeWeb3J -> to initialize web3j
— — — — — — — — — — — — — —
//to communicate with http client
HttpService gethHttpService = new HttpService(geth_url);
web3j = Web3j.build(gethHttpService);
— — — — — — — — — — — — — — -
7.2 withdrawFromContract -> to withdraw tokens using smart contract
— — — — — — — — — — — — — -
//create instance of smart contract and withdraw 10 tokens
SmartToken_sol_SmartToken greeter = SmartToken_sol_SmartToken.load(CONTRACT_ADDRESS, web3j, credentials, getGasPrice(), getGasLimit());
transactionReceipt = greeter.withdrawToken(recipient, tokentowithdraw).sendAsync().get(3, TimeUnit.MINUTES);
— — — — — — — — — — — — — -
7.3 depositToContract -> to deposit tokens using smart contract
— — — — — — — — — — — — — -
//deposit 10 tokens
SmartToken_sol_SmartToken greeter = SmartToken_sol_SmartToken.load(CONTRACT_ADDRESS, web3j, credentials, getGasPrice(), getGasLimit());
transactionReceipt = greeter.depositToken(recipient, tokentodeposit).sendAsync().get(3, TimeUnit.MINUTES);
— — — — — — — — — — — — — -
7.4 getTokens -> to get currently available tokens
— — — — — — — — — — — — — — -
//get tokens
SmartToken_sol_SmartToken greeter = SmartToken_sol_SmartToken.load(CONTRACT_ADDRESS, web3j, credentials, getGasPrice(), getGasLimit());
tokens = greeter.getTokens(recipient).sendAsync().get(3, TimeUnit.MINUTES);
— — — — — — — — — — — — — —
7.5 Build Andriod app and install
8. Run Blockchain nodes as given in section 1 to 5 of part2 of this article:
Smart Home Solution using Android and Blockchain:
9. Control Servo motor SG90
9.1 RPI and Servo motor connection
red wire — +5volt
brown wire — Gnd
yellow wire — signal
RPI connection
pin4 -> 5volt — red wire (better to use external battery.)
pin6 -> gnd — brown wire (connect gnd line to RPI gnd, if external battery is used)
pin 17 -> GPIO — yellow wire
9.2 Install pigpio:
sudo apt-get update
sudo apt-get install pigpio
sudo npm install pigpio
9.3 Create RPI client side app — smart_token_servo.js to control the servo motor based on number of tokens available in RPI account.
or
copy the code from:
if tokens > 1, rotate servo motor to 90degree to open the door
if tokens == 0, rotate servo motor to 0 degree to close the door
— — — — smart_token_servo.js — — — — —
// Interaction with GPIO
const Gpio = require(‘pigpio’).Gpio;
const motor = new Gpio(17, {mode: Gpio.OUTPUT});
// Interaction with Ethereum
var Web3 = require(‘web3’)
var web3 = new Web3()
// connect to the local node
web3.setProvider(new web3.providers.HttpProvider(‘http://localhost:8042'))
// The contract that we are going to interact with
var contractAddress = ‘0x0ce678aa4dabef4627fd351deeb35889e6c3d246’
// Define the ABI (Application Binary Interface)
var ABI = JSON.parse(‘[{“anonymous”:false,”inputs”:[{“indexed”:true,”name”:”_from”,”type”:”address”},{“indexed”:false,”name”:”_value”,”type”:”uint256"}],”name”:”OnValueChanged”,”type”:”event”},{“constant”:false,”inputs”:[{“name”:”recipient”,”type”:”address”},{“name”:”value”,”type”:”uint256"}],”name”:”depositToken”,”outputs”:[{“name”:”success”,”type”:”bool”}],”payable”:false,”stateMutability”:”nonpayable”,”type”:”function”},{“constant”:false,”inputs”:[{“name”:”recipient”,”type”:”address”},{“name”:”value”,”type”:”uint256"}],”name”:”withdrawToken”,”outputs”:[{“name”:”success”,”type”:”bool”}],”payable”:false,”stateMutability”:”nonpayable”,”type”:”function”},{“constant”:true,”inputs”:[{“name”:”recipient”,”type”:”address”}],”name”:”getTokens”,”outputs”:[{“name”:”value”,”type”:”uint256"}],”payable”:false,”stateMutability”:”view”,”type”:”function”}]’)
// contract object
var contract = web3.eth.contract(ABI).at(contractAddress)
// display initial state
showStatus()
// wait for an event triggered on the Smart Contract
var onValueChanged = contract.OnValueChanged({_from: web3.eth.coinbase});
onValueChanged.watch(function(error, result) {
if (!error) {
showStatus()
}
})
// control the door according the value of the token
function showStatus() {
web3.eth.defaultAccount = web3.eth.coinbase
var token = contract.getTokens(web3.eth.defaultAccount)
if (token > 1) {
//open the door
pulseWidth = 1500;
motor.servoWrite(pulseWidth);
} else {
//close the door
pulseWidth = 500;
motor.servoWrite(pulseWidth);
}
}
// release process
process.on(‘SIGINT’, function () {
greenLed.unexport()
redLed.unexport()
button.unexport()
})
— — — — — — — — — — — — — — — — — — —
9.4 Run the client application
cd ~/RPI/SmartToken
node smart_token_servo.js
10. Run Android application
Show the face
Now the face is recognized by the Android application
Observe door is opened
Don’t show the face
Observe door is closed
Note: 10seconds delay is introduced between any state change from Open to Close or Vice-versa.
11. Demo
The demo of this project is given in the link:
Source code:
References
1.Problems calling Smart Contract function in Java using Web3j
2.Raspberry Pi Servo Motor Interface | How to Control a Servo Motor using Raspberry Pi?
3.Raspberry PI Node.js Modules
https://www.w3schools.com/nodejs/nodejs_raspberrypi_components.asp
4.pigpio
https://www.npmjs.com/package/pigpio
5.Raspberry Pi Servo Jitter
Contact details
Mail: anbunathan.r@gmail.com
Website: http://digitranssolution.com
