Friday, 14 April 2017

Hackster, YouTube Channel and Safe UsbKey Project

Hi,
 if you are a reader of this blog.. I'm still alive :)

I've built in the last year, a lot of project (#alexa, #arduino mkr1000, #arduino 101, #linkit smart etc), you can find on my hackster page: https://www.hackster.io/crazycoder1999/

I'm starting to make some video on my YouTube Channel .

Right now, I have a side project called Safe UsbKey Project : here is the playlist of the first videos.
I will stream live today a new episode on how to create a build environment for the board.

The idea of Safe UsbKey Project is to build a sort of filter/proxy between computer and usb key in order to protect computers against viruses and other attacks.
The project use the arietta g25 from acmesystems.it.

Thursday, 15 September 2016

Testing an Alexa Skill

If you had in these weeks read my previous post (one and two) you may have took a look at the code
of the skill I wrote: https://github.com/crazycoder1999/EuroSoccer_Alexa_Skill/
This post will show some "tricks" I used to minimize the problems related on testing an Alexa skill.

BlackBox
The black box testing approach of an alexa skill is really complicated, it literally means: try vocally, all the possible inputs that your skill can (and can't) manage.
You can proceed in this way, of course, but you should NOT, it is an insane amount of work.

For the black box testing you can use a compatible echo device or a echo simulator: https://echosim.io/



Advice #1: Separate the code
The first advice I can give is to separate the business logic from the presentation: separate the code
that exposes the Intents of your skill from the function that answer to those Intents.

If you read the code of my skill you will see that index.js contains just the intents and the imports of
the modules.
The main code of the skill is in EuroUtils.js.
This simplify debugging and it helps to write unit testing code as you can see in test.js.

Instead of a single file, like test.js for unit testing, it is better to use one of the available Unit Testing
framework for nodejs.
If you have nodejs installed you can test EuroUtils.js launching node test.js inside the src folder.

Advice #2: Utterances generations
Another thing I didn't liked so much in Alexa, it is the way you have to create utterances.txt, because:
- it is too confusing
- it is hard to remember the logical link between Utterances and Intents

The more complex is your skill, the more complicated it becomes to keep track of them.
In order to improve the logical link between the two things I invented a new file, an xml, with some tags that helped to explain which Intents answers to a set of utterances.
There is an example here: utterances_groups_sample.xml

I than created a simple python script to extract and print on console: all the utterances to submit for the skill.

Have you got any other advice?
Write in the comments!

Thursday, 4 August 2016

Alexa, explain me how Skill works


(Continue from the 1st Post)
Amazon's voice assistant is called Alexa and it can run on Amazon Echo, Tap, Dot, FireTV and Fire Stick.
Amazon allows users to improve Alexa capabilities with the installation of 3rd party "Skills".
Some Skills are simple enough to be developed, even with very basic programming background.

Amazon is doing a very good job to teach developers how to build skills: code on github, webinar, tutorial, podcast, documentation and swaaaag!

Thinking about what people can ask

The developer of a Skill need to:
- write a list of possible questions a user can ask to Alexa for the skill
- develop a list of operations that answers for each possible questions to Alexa

Each operation can be connected to multiple questions, instead, for each question there is only one operation that can answer.

The operations are called Intents, meanwhile the questions are  called utterances.

Amazon take care of all the rest: recognize questions and parameters in user's questions and connect it to the right operation. The operation is executed and the response is sent back to the user.

Responses can be:
- voice responses
- voice and text/image: the contents is available on a mobile app for android / ios

The role of the mobile app is to give the user a way to install,delete and search for skills, give some informations on how to use a specific skill.
The way Alexa works, sometimes remeber the assistant of the movie "Her".



The mobile app helps the user to find the correct questions for the skill, but a very well done skill is the one in which for a specific intent the developer generated a very exhaustive list of possible questions.

Code on Amazon Lambda

Even if you can create and host your skill somewhere on internet with specific constraints and guide lines in any language you want, the easy way to code a skill, it is to use Amazon Lambda.

Most of the code example on github released by amazon are lambda-ready and built with javascript/nodejs, so with very few changes your new skill/bot is online.
If you are new with bots, I highly encourage to start with lambda so you can put all your efforts on the voice experience.

In the next post I will talk about testing and deployment of the EuroCup skill.
The code is available on my github here.

Monday, 1 August 2016

Attack of the Bots

Bots are a new tech trend in 2016.
Amazon,Facebook,Google,Microsoft,Apple all important companies announced bots support/integration in their products.

Of the all solutions/platforms the most interesting one for me, initially, was Amazon Alexa: because it is an expandable voice assistant.
(Hey Jeff Bezos! I'm still waiting the possibility to buy an Echo in Europe!).
I decided to build one bot, that works for 2 platform: facebook and amazon.

This and the posts that follows are a description of thoughts and choices I made to build it.

I didn't understand why the bots took so much attention on those tech companies until I started to figuring out some advantages over a traditional APP:

User advantages:
1 - Integration in known software: most of the solutions proposed are integrated in app or service known.. facebook messenger, skype, siri. Users already know them: No learning curve, they can rapidly start to talk with a bot with no need to learn and remember actions on user interface.
2 - Natural Interaction: everyone knows how to text messages and talk. This make bots attractive also for old people: it is not important if they can recognize if they are talking with a fake or real person, the important thing is if they can have the answers they are looking for.
3 - No installation/space needed on the device: no need to install anything on their device. No need to upgrade.
4 - Same uniform experience across devices

Developer adavantages:
5 - Userbase Potential: deploying something that become part of a known app (like facebook messenger) improves the link between the real user and the owner of the bot. In the case of facebook each Facebook Page(the page of a shop for example) can have its own bot that replies to a client. It is not just an anonymous user who watch a website, but a user with its information... Very valuable information if well used.
6 - One language for many platforms: web, web-mobile, desktop, mobile . No need to bother with accounts,login, oauth, tecnologies problems and so on!

Of course, Bots can't replace apps that make deep use of hardware on our device games , multimedia applications, photo application, complicated interaction apps and many other categories.
The challenge is also with web page ... but don't forget the interaction model.

Project Euro 2016
At the end of May, I decided to build a bot that gives to users, information about the European Football Cup 2016, informations like
- matches
- results
- teams
- groups

The bot were available for Amazon Alexa (it was published) and Facebook Messenger (it was not published).
The other important feature of the bot was to have a common data layer that were available for the 2 specific platform.



In the next posts I will explain the Amazon Alexa solution.

Thursday, 28 July 2016

Automator (OSX): 2 handy ideas


Automator is a tool shipped with OSX: it allows user to create simple applications that automates manual task. You can find some guide on how to use it, like this one.
Even if it is possible to record and save actions mad on the computer, I recommend to create the script manually.

In the last year I used this application for 2 scenarios.

Automatic launch applications for a Demo

When you create a demo for a presentation, it is important to be prepared and make no error: time is very important!
In november I did a speech with a final "demo".
The demo used a series of applications launched from the terminal:


The explanation of the above workflow:
  • opens a terminal
  • executes a command on the terminal (using keystrokes)
  • opens a new tab thanks to keypress [command + t]
  • executes another command
  • opens a third tab
  • keystrokes another command
  • opens the last tab, change the directory, launch a node script

Automatic Screenshots

Recently I subscribed an e-learning service with slides, available only online.
I wanted to have the same slides offline.
Thanks to automator it was possible to:
  • record the action to: select a window(the browser) and click on it. A click move the slide to the next one.
  • use screencapture to save in a predefined directory a screenshot of the entire screen. (using a bash command to build a unique filename each time).
  • repeat the step a variable number of times.
The script follows:



Thursday, 28 April 2016

One Year with Pebble (Time): a user and a developer perspective

The smartwatch "revolution" didn't happened as expected by the industry.

The sales numbers tells that Samsung, Google or Apple have conviced enough people.
Smartwatchesare perceived like new toys, instead of useful things and, often the disadavantages of having one are more than the advantages:
- ridiculous battery life
- no killer apps
- ridiculous apps (sending hearths or drawing to someone...)

and they are also expensive.

(Another interesting review from engadget about apple watch one year on)

With a lot of skepticism, last year I decided to buy one, just to understand the potential of these devices in everyday life.

Since, I found frustrating to charge my watch every night, I decided to become a bacher of the new pebble time, because it guarantees (in some conditions) 7 days of battery life, display always on, waterproof, and imho it is designed as a watch first, instead of a wrist-phone.

I'm not a typical smartwatch user.. not the one for who a smartwatch is designed for.
A smartwatch is useful if it can cover some microtasks related to "TIME" like:
- calendar notification
- alarms
and it is important that if it can work most of the time without the dependency from the smartphone.

I don't care about app notifications, neither of app integration.
I like the possibility to develop my own apps and expand watch functionality with apps.

BTW, an honorable mention is for the stock Health app, it tracks my daily sleep and daily steps: it is an addictive functionality and useful.

I use the smartwatch also in the sea, during surf sessions and this is a plus, because I can leave my phone inside the bag on the beach and receive call notifications.

Watchfaces are a nice idea, to change the look to the device but, you have to choose wise if you want to preserve battery life, because some "cool" watch faces can drain your battery faster.

Developing a WatchApp: "Work Time"
One thing I do when I arrive at work, it is to keep track of the arrival time: in order to calculate when I can leave the office.
This is a repetitive task and it can easily covered by my pebble!

I developed an app "Work Time" that vibrates when it is time to go home.
Create an app for a limited device it is an interesting challenge: inputs, cpu, memory, screen are limited.

Work Time is composed by these screens:
- Estimated Exit: It show you at what time you can leave
 

The app is configured using these 3 screens in sequence:
- Check In: the time of the arrival. Default is the current time.
- Work Time: how much time you work
- Break : how much time is your break


When your settings are done, you can close worktime, it will notify you with a vibration when you can go home and a screen like this one:
 

The pebble documentation is better than other platform I developed for... but creates a watchapp for pebble is not that easy due to:

- small community around the subject
- hard debugging: logs based
- cloud.pebble.com: online based editor and sdk.. sometimes it is a limit
- C language: good for optimization, bad for complexity


Btw, if you are interested, the code for WorkTime is here: https://github.com/crazycoder1999/worktime

Saturday, 28 November 2015

When an ESP8266 encounter MQTT and an accelerometer

This is a project I built for a presentation at http://milan2015.codemotionworld.com/ on November 21st.
This idea use ESP8266 (esp04), Nodemcu, MQTT and an accelerometer (adxl345 / sparkfun) by I2C.
It is an IOT project.
This project started as a "drone project" but the original idea changed during time.. by the way, helped me to do:
  • some experience with MQTT and I2C
  • see how much I can push the Esp8266
It was an interesting challenge!
Check the slides on my previous post to learn more about the project.
In my final demo I had:
Feel free to write me if you want to repeat the project and need helps.


Tuesday, 24 November 2015

ESP8266 and IOT talk at Codemotion Milan 2015

I was a bit far from the blog.. but it was because I was preparing a talk for

I built a demo using nodemcu, esp8266, mqtt, mosquitto, nodejs and adxl345.
Shortly: reading raw data from accelerometer (x,y,z) and send that data with MQTT through wifi.

More info coming veeeery soon..with sourcecode!

Wednesday, 22 July 2015

My adventure with ESP8266 (2/2): flash it!

Introduction 
The first part is here and it is about:
  • hardware
  • software
  • firmwares
  • nodemcu
It contains important information to understand this part.

Wirings
This table show how to connect the hardware together:


Pay attention to GPIO0, you need it connected only for flashing.

Flashing!
It's time to flash the device with a new firmware using esptool.
I decided to flash a nodemcu firmware.
Connect the GPIO0 to GND and than power on the device.
On my installation on OSX I did this command to flash the device:

python esptool.py -p /dev/tty.usbserial write_flash 0x00000 nodemcu_21072015.bin

If everything is ok, you should see this message:


Power off the esp8266 than remove the GPIO0 connection and than power on the chip again.

Talk with ESP8266
Now, it's time to talk with the chip.
I use kermit with this .kermrc configuration:

set carrier-watch off
set line /dev/tty.usbserial
set speed 9600

than you can press "c" to connect and typing print ("hello"); you should see the device responding hello back.
Now you can test some commands.

If you have downloaded the luatool, you can write your programs in a text file with lua extension and upload on the board with

python luatool.py --port /dev/tty.usbserial --src main.lua --dest main.lua --baud 9600

Lua files
Now, you can connect to the esp8266 and type dofile("main.lua"); to run the program.

If the file you are loading on the device, it is called init.lua the esp8266 at every boot will load it: pay attention, a bad init script can block the device forever... The only solution in this case is a re-flash of the device.

If you have any question, feel free to ask in the comments section.
If you like the article please share it! :)

my wirings :)



Tuesday, 21 July 2015

My adventure with ESP8266 (1/2)

Introduction
This article will explain all the steps I did to succesfully use the ESP8266 chip.
The ESP8266 is a an incredible low cost chip (< 5$!) to enable wifi communication for your project.
It is capable of:
  • create a wifi network: a wifi hotspot
  • connect to existing wifi hotspot
  • programmable digital pins: with I2C or SPI support!
I think this is ideal for many IOT project, at least in a prototype stage.
The chip is pretty new but thanks to its "open" nature and its cost, there are already some alternative firmware available.
This guide is tested for OSX Yosemite, but you can find similar tools also for Linux and Windows.
You can find different variants of ESP8266, but I think this tutorial will work equally with all the versions.

My version is the "04":
isn't it too cute ??? :D

My Hardware
  • CH340G USB to Serial (TTL) converter
  • ESP8266: I have the version 04
  • A stable 3.3v power with enough ampere to power the chip: I used a step down converter from 5V to 3.3V, the LM2596 calibrated with a multimeter and connected to a 5V/2A phone charger
  • a computer
Software
  • driver for CH340G here
  • kermit-c or any serial terminal software (minicom for example). For OSX
  • a firmware to flash: read below
  • esptool.py here
  • luatool here: valid only with nodemcu firwmare
Firmwares
The default firmware of the chip can be outdated.
This firmware exchange data with AT command on serial line.
A list of commands is here, but commands can change in every firmware.
This firmware is good if you need to connect the ESP8266 to other components (for example an arduino).
My esp8266 comes with a firmware not capable of talk with AT commands in any f*****g way, it started as an hotspot (see image) but I was not able to talk with at any serial speed.

BTW, the hotspot check is a good way to control if the ESP8266 is correctly connected


I flashed a new updated version of the official firmware.
The default firmware is splitted in 4 .bin files: I have to power off and than power on the chip after each flash.
I than tested the AT command with kermit-c, pressing CTRL+J in the terminal after every typed command.
As far as I know, It is possible to create custom programs using the EspressIf SDK and C++.

NodeMCU
I decided to test another firmware and the choice was the nodemcu firmware.
This firmware lets you write scripts in LUA language; it's a new language for me, but the syntax is pretty simple.
The advantage here are:
  • upload scripts, without reflashing the firmware
  • the availability of a powerful and simple API with a good documentation

[the second part is available here]

Monday, 15 June 2015

Android + Gradle: Build different APKs with different dependencies

Introduction
Recently, I switched my projects, to Android Studio.
Two years ago when Google announced the new IDE, I was skeptic: android studio was buggy and gradle a new complicated tool to learn.
Today I'm an happy user of android studio.

In this tutorial, I will talk about my experience with gradle and some useful customization I made on the build process of my app, called Qoffee.

The problem
Qoffee is available on Play Store and Amazon App Store and allows users to find the best coffee in town.
One of the last features I added, it was the support for Android Wear: now it is possible to search coffees from the smartwatch!

With the new features:
- the new apk is passed from 1.2 mb to 3.9 mb
- the new apk contains also the apk to install into the smartwatch
- there is a new Service, precisely a WearableListenerService. This service is the one that talks with the watch.

New classes/features were not useful on the apk for other app store (eg. amazon).
In the new solution, it is now possible to automatically build 2 separate apk:
- one for play store: with all the features (size: 3.9 mb)
- one for other store: without android wear and google play services support (size: 1.2 mb)

Remove unused files: Separate the Code and merge manifests
I searched a way to create 2 different builds for my app, in order to create a custom Manifest and to remove the WearableListenerService from the non-play store apk.
In the src folder of my project on android studio I created 2 new “Java Folder”:
- playstore
- genericstore

Create a new Java Folder


New project tree


The original folder for my project was called “main”.
I created 2 other AndroidManifest.xml files, and placed them in the new folder playstore and genericstore like this:
- AndroidManifest.xml in main folder contains common data for both apk: eg. all the common activities
- AndroidManifest.xml in playstore folder contains only informations specific to the play store apk: eg. reference to the WearableServices class, keys for playstore services
- AndroidManifest.xml in genericstore folder contains information for non-playstore apk: mainly, empty

See the differences in the following image:

Differences between Manifest

After this operation I created two product flavors in the gradle file of my app like this:

android {
     compileSdkVersion 20
     buildToolsVersion '20.0.0'
     productFlavors {
          playstore{
               applicationId "com.andrea.degaetano.coffelover.playstore"
          }
          genericstore{
               applicationId "com.andrea.degaetano.coffelover.genericstore"
          }
     }
… … (other unchanged configurations here..)
}

The new build.gradle file generated 4 new “Build Variants” in Android Studio.
This new menu allows the developer to select which kind of build generate:

All my modules.. and the selected variant
you can change the selected variant on any module

At this point, if you try to export the apks, the genericstore apk fails, because the android wear apk is present in the genericstore apk and the package of the genericstore is different from the one in the android wear apk.
Not only, but the genericstore apk is always 3.9mb.

In order to remove the dependency from android wear in the generic apk, I have changed the dependencies of my application from:

dependencies {
     compile project(':parseLoginUI')
     compile 'com.android.support:support-v4:20.0.0'
     compile 'com.google.android.gms:play-services-wearable:7.5.0'
     compile 'com.android.support:appcompat-v7:20.0.0'
     compile files('libs/android-async-http-1.4.6.jar')
     WearApp project(':qoffeewear')
}

to

dependencies {
     compile project(':parseLoginUI')
     compile 'com.android.support:support-v4:20.0.0'
     playstoreCompile 'com.google.android.gms:play-services-wearable:7.5.0'
     compile 'com.android.support:appcompat-v7:20.0.0'
     compile files('libs/android-async-http-1.4.6.jar')
     playstoreWearApp project(':qoffeewear')
}

Important: The package of the android wear app is the same of the playstore variant: com.andrea.degaetano.coffelover.playstore

Thursday, 28 May 2015

React.js at Mobile Developer Day #2 Genova

These are my slides about React.js

It was just an introduction to React :)

Friday, 15 May 2015

Adding USB to Ethernet to Arietta

Introduction
Arietta is a low cost linux embedded module: http://www.acmesystems.it/arietta I already talked about it, here:
This post talks about adding a usb to ethernet adaptor for arietta.
The adapter is based on asix hardware, you can find it in various shapes:


Modding the Kernel
My build enviroment is based on a debian linux, created with vagrant.
There is this very good tutorial on acmesystem website: http://www.acmesystems.it/compile_linux_3_16 on how to compile a new kernel for the board.
Instead of following the tutorial untill the end, when you reach "make ARCH=arm menuconfig"you have to enable the "Multi-purpose USB Networking Framework" and select the appropriate ethernet adapter:


You can find the section under Device Drivers / Network device support / USB Network Adapters.
After you have rebooted with the new kernel you can plug the adapter and see if the device is recognized typing "dmesg".

My ethernet device is recognized as "eth0", in order to configure it with dhcp you have to edit /etc/network/interfaces on arietta and insert these lines at the end:

auto eth0
iface eth0 inet dhcp
 
Plug the ethernet cable, reboot the board and... done!







Friday, 10 April 2015

Parse and Android: my slide for DroidconIT 2015

Yesterday, I had a talk about Android and Parse, in Turin.
The talk was inside a barcamp session.
The slide are here:



More articles on parse.com and android will be available on the blog in the next months :)

Thursday, 2 April 2015

Number / Money Picker Dialog in Android

I needed to create a money picker dialog for one of my apps called Qoffee.
This money picker is used to signal the price of a coffee in a bar.
Here is the dialog:


To build this dialog I created an AlertDialog with a Custom Layout.
First of all you need to define the layout with a classic xml layout file:


< LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"  >
    < TextView
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:padding="10dp"
        android:textSize="16sp"
        android:text="@string/bar_change_price"/>

    < LinearLayout
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:gravity="center">
       
    < NumberPicker
        android:id="@+id/euro_picker"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center" />
   
     < TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:padding="5dp"
        android:textSize="20sp"
        android:text="@string/point_symbol"/>
   
    < NumberPicker
        android:id="@+id/cent_picker"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center" />
     < TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:padding="5dp"
        android:textSize="20sp"
        android:text="@string/euro_symbol"/>
   
    </ LinearLayout>
</ LinearLayout>

In my case this is made by:
  • 1 textview for main dialog description
  • 2 number pickers: 1 for euro and 1 for cents
  • other descriptive textview: for euro and point symbols

If you pay attention, you see I didn't defined any buttons.
The bottom buttons are defined as part of the AlertDialog: you don't have to define it in the layout.

The code to show this dialog follows:

AlertDialog.Builder builder = new AlertDialog.Builder(getActivity());
LayoutInflater inflater = getActivity().getLayoutInflater();

View theView = inflater.inflate(R.layout.number_picker_dialog, null);
//I define the dialog and I load the xml layout: number_picker_dialog.xml into the view

final NumberPicker unit_euro = (NumberPicker) theView.findViewById(R.id.euro_picker);
final NumberPicker cent = (NumberPicker) theView.findViewById(R.id.cent_picker);

// I keep a reference to the 2 picker, in order to read their properties for later use
                  
builder.setView(theView)
.setPositiveButton(R.string.accept_price_change,new DialogInterface.OnClickListener() {
    @Override
    public void onClick(DialogInterface dialog, int which) {
            Log.d("DBG","Price is: "+unit_euro.getValue() + "."+cent.getValue());
    }
}).setNegativeButton(R.string.reject_price_change, new DialogInterface.OnClickListener() {
    @Override
    public void onClick(DialogInterface dialog, int which) {
                      
    }
});

// I define 2 default buttons, with 2 strings (accept_price_change and reject_price_change) and their behaviours

unit_euro.setMinValue(0);
unit_euro.setMaxValue(3);

// I define the range for the first numberpicker.

String cents[] = new String[20];
for(int i = 0;i < 100; i+=5) {
    if( i < 10 )

       cents[i/5] = "0"+i;
    else
        cents[i/5] = ""+i;
}

cent.setDisplayedValues(cents);
//I create the range of the possible values displayed in the second numberpicker.

cent.setMinValue(0);
cent.setMaxValue(19);
cent.setValue(0);          
   
//I configure the possible values of the second picker

builder.show();

//Finally, the alert is showed!

Wednesday, 25 February 2015

Security Webcam: (Improve) Uploading Photo to Dropbox with a Linux Board


Recently I bought a used webcam Microsoft Lifecam HD3000 on ebay.
I was interested in this device because it is Linux compatible and it can be used with one of my linux boards: Arduino Yun.



I was interested to use the Webcam, the Yun and a Pir Sensor to make a rudimental security camera system for my home.

If you are interested in this type of activity, you can find a lot of material, just "googling" (and of course you can use another linux board, like a raspberry pi):
What I didn't like from these projects is the way they execute actions from the arduino side.

The components of my finished project are very similar to the first link from adafruit.com: a webcam, a pir sensor and the arduino yun.

The events on the Yun, Arduino side
The example that I've found works like this:
  1. wait for move event (eg. from pir sensor)
  2. use Process and call fswebcam to get a photo from the webcam
  3. use Process and call a python script to upload a photo on dropbox
  4. ... do something ...
  5. go back to 1.
Everytime in a yun's sketch you will use Process, you are blocking the execution of the arduino part until the command is completed.
As long as the step 3 is completed, it is impossible to capture another photo.
Some samples use Temboo for dropbox, but I don't like it: it makes the sketch more complicated to mantain and the Dropbox Api is easy to use.
Since I didn't want to wait the 3rd step or use Temboo api on my system, I have wrote some code.

My solution: Arduino part
In my solution the arduino part work like this:
  1. wait for move event (From pir sensor)
  2. use Process and call fswebcam to get a photo from the webcam saving it, in a temporary directory
  3. use Process to move the photo to a destination directory where will be picked by the linux part
  4. go back to 1
The step number 3 is needed to be sure that the photo is totally written before the upload on dropbox.
Now, I have again 2 call to Process... but move a photo of less than 100 kbyte from a directory to another is faster than uploading the same photo to dropbox.
The arduino side, it is indipendent from the dropbox upload: this is a job for the linux side.
(if you want the arduino's sketch, just ask in the comments).

Linino part
I wrote a python script upload_to_dropbox.py in /root, that is able to continually check the photos's destination directory and when it finds one new, it uploads it on my dropbox account and on a succesfull upload, it deletes the local photo from the yun.
For setup of python and dropbox library on arduino yun I have followed this tutorial.
What I have changed from it, it is the sketch and the python script.

The python script is here, just edit it with your generated dropbox key.

 import dropbox  
 import glob   
 import os  
 import json  
 from time import sleep  
 photodir = '/mnt/sda1/arduino/www/';  
 client = dropbox.client.DropboxClient('DropBoxGeneratedKeyGoHere')  
 #starting main loop here  
 while(1):  
      toupload = glob.glob(photodir+'*.jpg');  
      try:  
           for fname in toupload:  
                print "Uploading... " + fname;  
                f = open(fname, 'rb');  
                response = client.put_file(fname[len(photodir):], f);  
                print "uploaded" + fname[len(photodir):];  
                print response['bytes'];  
                if(response.has_key("bytes") and response['bytes'] > 0):  
                     os.remove(fname);   
           sleep(0.3);  
      except:  
           sleep(2);  

In order to start the script automatically on each boot of the Yun, you need to edit /etc/rc.local adding:

export PYTHONPATH=/mnt/sda1/python-packages
python /root/upload_to_dropbox.py &





Sunday, 11 January 2015

Extracting data generated in a Facebook Group

Introduction

I do Parkour since august of 2007 and I have also created with other people in town a Facebook Group.
This facebook group is used by practitioners and beginners in order to find other people to train with, make questions,  post photos and links on parkour but also schedule training appointment.
I was curious to know which was the spot/the place, where practitioners have played most activities in the 2014.



Facebook Data: Graph Api Explorer
To dump all the post of facebook of my parkour group, I started with the documentation at developer.facebook.com.
The API is called Graph Api, the documentation at facebook.com is good enough done: https://developers.facebook.com/docs/graph-api/using-graph-api/v2.2

One of the first tools to use to test the facebook API is Graph Api Explorer https://developers.facebook.com/tools/explorer/145634995501895/

Graph API Explorer (GAE) is useful to test facebook api and to generate a valid token for your tests/scripts, just press "Get Access Token" button on the user interface.

The first step with GAE is to generate a valid token for your logged user with the permissions requested for your queries: in my case user_groups is mandatory.
I've tested then some queries:
  • "me" show data of the current logged user
  • "me/friends" show data about the friends of the logged user
  • "me/groups" show data about the groups of the logged user
Graph Api Explorer, the response (cutted) on a user data
The data is returned in JSON format and every returned object: person, comment,  status, group...etc  have an own id.
The "me/groups" is useful to find the group-id of the group to later analysis.
So, simply putting the group-id and pressing then submit, it will show the information of the group; if you need to fetch all the feeds of the group you need to insert "group-id/feed".

Pagination and period of interest
It is not possible to fetch all the posts of a group with one query: results are divided in pages.
Every succesful json response contains a "paging" section

  "paging": {  
   "previous": "https://graph.facebook.com/v2.2/34343/feed?since=1420799539&limit=25&__paging_token=enc_AexSv5Apv-nbkvNZ",  
   "next": "https://graph.facebook.com/v2.2/23ddsds/feed?limit=25&until=1417117943&__paging_token=enc_AezUcra6A2_dsSv5A"  
  }  

This section contains 2 links one to query the next results and the other to previous results.
Another important parameter to consider is the "until" parameter (or since), it needs to be used to return all the post until a certain period expressed as Epoc Timestamp.
In order to get all the post of a particular year:
  • I fetch the results recorded until 1/1/2015 00:00:00
  • I use the "next" record until I have a first record with created_time  == 2013
  • all the data is saved in a file for later analysis
  • between one fetch and another, it waits 5 seconds, in order to be safe from api rate limit

The script
To use the following script you need a python enviroment with this library installed: https://github.com/pythonforfacebook/facebook-sdk (on a mac, download it, then sudo python setup.py inside the directory)
It follows the script to analyse all the 2014's posts of a particular group; you need to edit:
  • value of the token variable
  • value of the group_id variable
  • change the start_date_time variable
  • change the stopyear variable

 import time  
 import datetime  
 import facebook  
 import cPickle as pickle  
 token = "put your graph api key here you can have one with Graph API Explorer"  
 group_id = 'put your group id here';  
 start_date_time = '01.01.2015 00:00:00';  
 pattern = '%d.%m.%Y %H:%M:%S'; 
 stopyear = '2013';
 epoch = int(time.mktime(time.strptime(start_date_time, pattern)))  
 print epoch  
 #code control from epoch string... -> print (datetime.datetime.fromtimestamp( epoch ).strftime('%Y-%m-%d %H:%M:%S'))  
 action = group_id + "/feed?limit=25&until=" + str(epoch);  
 print action;  
 graph = facebook.GraphAPI(token)  
 urlLen = len("https://graph.facebook.com/vX.Y"); #to remove from every request.  
 goOn = True;  
 allRecords = [];  
 while (goOn):  
      profile = graph.get_object(action)  
      if 'data' in profile:   
           print 'lecit...';  
      else:  
           print 'problems found... rate limit?';  
           print profile;  
           break;  
      allRecords.extend(profile["data"]);  
      if stopyear in profile["data"][0]['created_time']:  
           print "stop";  
           goOn = False;  
      else:  
           print "not ready..Go On."+str(len(allRecords));  
           action = profile["paging"]["next"][urlLen:];  
      time.sleep(5);  
 with open('genovapkdata.fb', 'wb') as fp:  
   pickle.dump(allRecords, fp)  

Conclusions
At the end of the execution I will have a file (genovapkdata.fb) with all the record saved in a format readable by python with another script:

 import cPickle as pickle  
 data = "";  
 with open('genovapkdata.fb', 'rb') as fp:  
   data = pickle.load(fp)  
 print "Records: "+str (len (data));  
 countGovi = 0;  
 ....  
 for onerecord in data:  
      if 'message' in onerecord:  
           msg = onerecord['message'];  
           msg = msg.lower();  
           if 'created_time' in onerecord and '2013' in onerecord['created_time']:  
                print 'record not important';       
           elif 'govi' in msg:  
                countGovi = countGovi + 1;  
 ...  

I discard posts with created_time of 2013 again, because this time I'm analysing 1 post at time, and I'm also counting record with a particular word in it "govi"... of course I did other analysis.

It is possible now do some statistics like:
- know who is the main post-writer in the group
- do a top10 of the place where the people train most
- how many are the generated posts
- how many are the generated comments

Monday, 22 December 2014

Why every developer should attend an hackathon?

In the past 3 years I have joined at least 10 hackathons, most of them in my country(Italy).
This is me, explaining my idea in the last hackathon I took part: Techcrunch Disrupt Hackathon (London): http://on.aol.com/video/extendernote-at-techcrunch-disrupt--london-2014-hackathon-518470061

Hackathons are a great way for companies to find new ideas around their product and test it in different situations and contexts.
These events are a mashup of tecnologies and products, mixed together in a very short time: a situation not easy to reproduce differently and for a lot of companies not economically possible.

When the product is an API or a libraries, an hackathon is pure gold; they are a good investment with tiny effort, that helps companies to make blackbox testing and be able to validate: documentation, libraries, api and discovers problems of their product.
During an hackathons for 24-48 hours, hundreds of developers are testing company's product/api/libraries for "free".

It is not always a win for a company, some projects/products are just ignored by developers during an hackathons but at least a good way to promote a product.

Btw, there are a tons of good reasons also for developers to be part of an hackathon:
- Test your skills: during an hackathon you will need to put your best knowledge, trying to build a working prototype in a very short time. Every problems will keep you far from your objectives, forcing you to reduce the features you want. It's challenging.
- Networking:find people to create a collaboration (designers for example), talking with people of other places and discovering their way of thinking, that is  culturally different and then, it is interesting.
- Prices: good prototype, good pitch, good idea and a bit of luck, with those ingredients it is possible to win cool prices.
- Validate your ideas: do you think to have a good idea to show? don't you want to measure how good is it? It's the perfect situation: you validate ideas against strangers, not only friends.
- Learn new tecnologies from others of your team or improve your knowledge: I've tested ionicframework during an H this year, just to see how usable is, in a real situation (= not ready enough)
- Feeling home talk with other developers about geek's things: new products, libraries, technologies and ideas.
- Find a new job Never gone to an H for this, but it happens that some companies are there to seek new employes. They'll come to you asking for an interview: "what are you doing in your life? What's your job? Why don't you come and visit our companies?" and so on.
- Pitching speaking in front of a lot of people and explaining your idea. You need to summarize, make choices, be clear and convince others of your work in a short time. Plus: do it in a foreign language.
- Enjoy the experience: it is a devastating phisically experience but you will be happy at the end anyway

And remember to be prepared before the hackathon:
- read how many documentation you can, about the aim of the hackathon and the sponsors (also searching on stackoverflow problems with api/libraries)
- write down some ideas you will like to build there and be prepared to join others idea
- choose a platform/tecnologies that you will use during the H
- prepare your hardware: tablet,smartphones, smartwatch ...
- prepare your software: libraries, framework, examples, documentation

So if you are now convinced and you want to keep an eye on hackathons near you, check www.challengepost.com


Monday, 11 August 2014

Smart Charger: automatically backup your photos

[This is the second post about Arietta: the first is here]

The project
The idea behind this little project is to have a system that charge my phone and automatically backup the photo of my android phone while it is connected.

First of all, you need to attach an usb female adapter to Arietta and connect an usb (powered) hub to it: Follow this page

In order to get the photos on the android phone from Arietta you need to use MTP protocol and there are 2 ways:
a - install mtp-tools packages from repository
b - activate mtp filesystem

For my project the first solution is easier:  launch apt-get install mtp-tools

The solution b was better, because it allows you to mount the phone filesystem and copy files from it with normal linux commands like cp,rsync,mv.. but it's harder because you need to rebuild the arietta image.

External Usb Storage
The Arietta will copy my photos on an external storage instead of the internal sdcard of arietta because this helps to improve the life of the sdcard.

To automatically mount an external storage in a predefined directory:
- use blkid command to obtain an id that identify you external usb storage for the system
- change /etc/fstab and insert a line to specify where the storage mount point

My external usb storage is a simple pendrive formatted with fat32 with UUID FC4D-37AA.
At the end of /etc/fstab I've added this line to mount it on /mnt everytime is plugged in:

UUID=FC4D-37AA /mnt vfat defaults,umask=007,gid=46 0 0


The Backup Script ...
I've created a script called photobackup.sh that use some commands from mtp-tools package to copy and delete jpg photos from the phone:

#!/bin/bash

#check if external storage is present or abort
checkmount=`mount | grep /mnt | wc -l`
echo $checkmount;
if [ $checkmount -ne "1" ]; then
        echo "Usb External Storage not mounted...exiting";
        exit;
fi

echo "Usb External Storage Found";
mtp-files | grep .jpg | awk '{ print $2 }' > lista.txt
while read F ; do
    #there is enough space on device
    spacecheck=`df -h | grep /mnt | awk {'print $5'} | tr -d '%'`
    if [ $spacecheck -gt "95" ]; then
        echo "Not enough space left on device..exiting";
        exit;
    fi
    mtp-connect --getfile $F /mnt/$F # copy the file
    mtp-connect --delete $F          # delete the file from the phone
done

Explanation:

- it checks if Usb External Storage is mounted or it aborts the operation
- it creates a list of jpg files (supposing these are all photos) lista.txt


... and for every photo:
- it checks if there is enough space left on the Usb External Storage
- it reads the list of photos from lista.txt and it uses mtp-connect to copy file to the /mnt/ directory and it deletes the photo from the phone


..and the UDEV Rule!
In order to call photobackup.sh everytime you plug the phone to the hub you need a udev rule.

Create a script 98-my.rules with this content:
SUBSYSTEM=="usb", ATTR{idVendor}=="22b8", MODE="0666", RUN+="/root/photobackup.sh"

and copy it on /etc/udev/rules.d

Now everytime you attach the android phone to the hub, while the phone is charging, arietta copy your photos in the external pendrive, if present.

In the next article we will see how to add a led to monitor the status of the operation.

Friday, 8 August 2014

How to hack Clash Of Clans: Unlimited Shield [UPDATE]

Introduction
*updates on the bottom
I've played for over a year with Clash of Clans: a very nice mobile games.
Day by day, the game become more and more difficult to play without spending money.
It's the classical in-app purchase free game mechanism: pay to survive.


The hack
During the game you have a limited shield, that can save your village from attacks for others player for a max of 16 hours.
Of course you can extends the shield with money.
When you have no shield, the others people can't attack you if you are online and connected.
My first attempt was to put the display of the phone in a "never sleep" mode and leave the app open but this was not a working solution, clash of clans see that you aren't do nothing and it disconnects.

Later I was thinking about an hardware solution with arduino, that could simulate at least the swipes on the touch screen, something like this hack for another game: https://www.youtube.com/watch?v=SnUH6f_Mv8o but there was an easier solution.

UiTesting with UiAutomator
One of the android's tools is called uiautomator http://developer.android.com/tools/testing/testing_ui.html it is used to test UI on apps, so it is possible to write test/programs that simulate click, swipe and other actions on apps running on the phone.
What I have done, it has been write a simple uiautomator test that:

1 - it presses home
2 - it searches clash of clans in the list of apps
3 - it launches clash of clans
4 - it waits 15 seconds for the app to start
5 - it does swipes movement
6 - it waits some seconds
7 - it repeats from 5

I tested the app on my phone (no root required) and my village was opened for over 10 hours(see update #) without any disconnection and of course, none attack! :)


Update #1: 18 Aug 2014
Yesterday I posted on reddit this post, to understand the reaction of some players and what they are thinking about it.
On reddit I've discovered that after 6 hours of game, Clash Of Clans automatically disconnect the client for 5-10 minutes... but, my hack (I'm lucky) keep on try to reconnect during this period and at the end of this period it is correctly reconnected.

I have also discovered there are some hacks out there that works with rooted phone and works in similar way.