Camera App Testing Using AI and Appium
Environment
Windows 10
Tensorflow=1.1.0
python=2.7 (for running Appium)
python=3.8 (for running keras-data-augmentation)
global modules path = C:\Users\natha\AppData\Roaming\npm
appium = 1.19.1
node = 14.15.0
npm=6.14.9
test-ai-classifies=2.1.1
Pre-requisites:
Install Android SDK and configure ANDROID_HOME Install Java SDK and configure JAVA_HOME
Please go through the part 1 of this article:
https://anbunathanramaiah.medium.com/re-training-test-ai-classifier-using-google-colab-49b4e2a35bac
Install ‘Open Camera’ app
Step1:
To get images from ‘Open Camera’ app using test-ai-classifier
Refer: C:\Users\natha\AppData\Roaming\npm\node_modules\test-ai-classifier\build\lib\classifier.js
Create environment variables:
DEBUG_IMAGES = true
DEBUG_IMAGE_DIR = D:\images
Run Appium script downloaded from GitHub:
https://github.com/anbunathan/appium-examples/blob/master/Template1/src/test/java/com/example/appium/TestCameraAI.java
Now slices are stored under this directory (D:\images).
(Note: Once slices are created, delete the environment variables DEBUG_IMAGES and DEBUG_IMAGE_DIR)
Step2: Prepare Icons
Use the image slices created in Step1 to prepare icons.
1.size = 224x224(recommended. but 120x120 also working)
2.To convert to Grey scale image
https://onlinejpgtools.com/convert-jpg-to-grayscale
Select option->HDTV method
3.Keep white paper in the back camera while taking screenshot
4. Use slices provided by test-ai-classifier to create icons
Give appropriate name for these icons
(Note: Avoid space or special characters in the icon’s name)
Step3: Image data augmentation
Download code from Github:
https://github.com/anbunathan/appium-examples/tree/master/keras-data-augmentation
Add these icons under “input_images” folder
Generate augmented images using “generate_images_in_loop.py” in:
(D:\trial\appium_examples\keras-data-augmentation)
(Note: Minimum 500 images are recommended. Also add original image)
Observe subfolders are created under “generate_images” folder
Each subfolder contains these augmented images
(Note: the folder name will be icon’s name)
Step4: Add these folders under “training_images” folder in:
(D:\trial\appium_examples\classifier-builder)
Zip into training_images.zip file
Upload training_images.zip to google drive
Copy from google drive to colab
https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99
!wget — load-cookies /tmp/cookies.txt “https://docs.google.com/uc?export=download&confirm=$(wget — quiet — save-cookies /tmp/cookies.txt — keep-session-cookies — no-check-certificate ‘https://docs.google.com/uc?export=download&id=1dN0c4DeOIqUTGohBAUKA_b2JaZWdNdhF' -O- | sed -rn ‘s/.*confirm=([0–9A-Za-z_]+).*/\1\n/p’)&id=1dN0c4DeOIqUTGohBAUKA_b2JaZWdNdhF” -O training_images.zip && rm -rf /tmp/cookies.txt
https://drive.google.com/file/d/1dN0c4DeOIqUTGohBAUKA_b2JaZWdNdhF/view?usp=sharing
Step5: Re-train as given in
https://anbunathanramaiah.medium.com/re-training-test-ai-classifier-using-google-colab-49b4e2a35bac
Note: select training steps = 4000
Step6: Copy web model to test-ai-classifier model directory
Copy the contents of web_model folder
Copy the contents of this folder to the npm dependency install for ‘appium-classifier-plugin’ under the model folder
!zip -r web_model.zip web_model
download zip file to local PC
copy web_model contents from
web_model folder
to
model folder under Appium root directory
(Example: C:\Users\natha\AppData\Roaming\npm\node_modules\test-ai-classifier\model\)
after renaming tensorflowjs_model.pb to model.pb, weights_manifest.json to weights.json. group1-shard1of1 as it is.
Step7: Edit label.js
After re-training, edit label.js to include new labels under lib folders.
Example:
C:\Users\natha\AppData\Roaming\npm\node_modules\test-ai-classifier\build\lib
and
C:\Users\natha\AppData\Roaming\npm\node_modules\test-ai-classifier\lib
Note: Order of labels should follow directory order in training_images folder (alphabetical order)
(Ex: const LABELS = [“camera”, “camera7”, “cameraauto”, “cameraflash”, “cameralock”, “camerasetting”];)
Step8: Test new labels
Re-start appium server (appium -a 127.0.0.1 -p 4723)
— — — — — — — — — — — — — — — — — — — —
Create an Appium script to launch ‘Open Camera’ app:
capabilities.setCapability(“appPackage”, “net.sourceforge.opencamera”);
capabilities.setCapability(“appActivity”,”net.sourceforge.opencamera.MainActivity”);
Also, add script to click menu items of ‘Open Camera’ app:
driver.findElement(MobileBy.custom(“ai:camera7”)).click();
Thread.sleep(5000);
driver.findElement(MobileBy.custom(“ai:cameraauto”)).click();
driver.findElement(MobileBy.custom(“ai:cameraflash”)).click();
driver.findElement(MobileBy.custom(“ai:cameralock”)).click();
driver.findElement(MobileBy.custom(“ai:camerasetting”)).click();
driver.navigate().back();
— — — — — — — — — — — — — — — — — — — —
You can download Appium script from Github:
https://github.com/anbunathan/appium-examples/blob/master/Template1/src/test/java/com/example/appium/TestCameraAI.java
Right click ‘testngCameraAI.xml’ and run Appium script to click new labels
Step9: Demo
A demo of running this script is given under:
https://youtu.be/1RADHDJPwUQ
References
How to properly install test-ai-classifier plugin using appium for ui selectors?
https://stackoverflow.com/questions/65280383/how-to-properly-install-test-ai-classifier-plugin-using-appium-for-ui-selectors/65280741?noredirect=1#comment115409539_65280741
