Sunday, 11 January 2015

Extracting data generated in a Facebook Group

Introduction

I do Parkour since august of 2007 and I have also created with other people in town a Facebook Group.
This facebook group is used by practitioners and beginners in order to find other people to train with, make questions,  post photos and links on parkour but also schedule training appointment.
I was curious to know which was the spot/the place, where practitioners have played most activities in the 2014.



Facebook Data: Graph Api Explorer
To dump all the post of facebook of my parkour group, I started with the documentation at developer.facebook.com.
The API is called Graph Api, the documentation at facebook.com is good enough done: https://developers.facebook.com/docs/graph-api/using-graph-api/v2.2

One of the first tools to use to test the facebook API is Graph Api Explorer https://developers.facebook.com/tools/explorer/145634995501895/

Graph API Explorer (GAE) is useful to test facebook api and to generate a valid token for your tests/scripts, just press "Get Access Token" button on the user interface.

The first step with GAE is to generate a valid token for your logged user with the permissions requested for your queries: in my case user_groups is mandatory.
I've tested then some queries:
  • "me" show data of the current logged user
  • "me/friends" show data about the friends of the logged user
  • "me/groups" show data about the groups of the logged user
Graph Api Explorer, the response (cutted) on a user data
The data is returned in JSON format and every returned object: person, comment,  status, group...etc  have an own id.
The "me/groups" is useful to find the group-id of the group to later analysis.
So, simply putting the group-id and pressing then submit, it will show the information of the group; if you need to fetch all the feeds of the group you need to insert "group-id/feed".

Pagination and period of interest
It is not possible to fetch all the posts of a group with one query: results are divided in pages.
Every succesful json response contains a "paging" section

  "paging": {  
   "previous": "https://graph.facebook.com/v2.2/34343/feed?since=1420799539&limit=25&__paging_token=enc_AexSv5Apv-nbkvNZ",  
   "next": "https://graph.facebook.com/v2.2/23ddsds/feed?limit=25&until=1417117943&__paging_token=enc_AezUcra6A2_dsSv5A"  
  }  

This section contains 2 links one to query the next results and the other to previous results.
Another important parameter to consider is the "until" parameter (or since), it needs to be used to return all the post until a certain period expressed as Epoc Timestamp.
In order to get all the post of a particular year:
  • I fetch the results recorded until 1/1/2015 00:00:00
  • I use the "next" record until I have a first record with created_time  == 2013
  • all the data is saved in a file for later analysis
  • between one fetch and another, it waits 5 seconds, in order to be safe from api rate limit

The script
To use the following script you need a python enviroment with this library installed: https://github.com/pythonforfacebook/facebook-sdk (on a mac, download it, then sudo python setup.py inside the directory)
It follows the script to analyse all the 2014's posts of a particular group; you need to edit:
  • value of the token variable
  • value of the group_id variable
  • change the start_date_time variable
  • change the stopyear variable

 import time  
 import datetime  
 import facebook  
 import cPickle as pickle  
 token = "put your graph api key here you can have one with Graph API Explorer"  
 group_id = 'put your group id here';  
 start_date_time = '01.01.2015 00:00:00';  
 pattern = '%d.%m.%Y %H:%M:%S'; 
 stopyear = '2013';
 epoch = int(time.mktime(time.strptime(start_date_time, pattern)))  
 print epoch  
 #code control from epoch string... -> print (datetime.datetime.fromtimestamp( epoch ).strftime('%Y-%m-%d %H:%M:%S'))  
 action = group_id + "/feed?limit=25&until=" + str(epoch);  
 print action;  
 graph = facebook.GraphAPI(token)  
 urlLen = len("https://graph.facebook.com/vX.Y"); #to remove from every request.  
 goOn = True;  
 allRecords = [];  
 while (goOn):  
      profile = graph.get_object(action)  
      if 'data' in profile:   
           print 'lecit...';  
      else:  
           print 'problems found... rate limit?';  
           print profile;  
           break;  
      allRecords.extend(profile["data"]);  
      if stopyear in profile["data"][0]['created_time']:  
           print "stop";  
           goOn = False;  
      else:  
           print "not ready..Go On."+str(len(allRecords));  
           action = profile["paging"]["next"][urlLen:];  
      time.sleep(5);  
 with open('genovapkdata.fb', 'wb') as fp:  
   pickle.dump(allRecords, fp)  

Conclusions
At the end of the execution I will have a file (genovapkdata.fb) with all the record saved in a format readable by python with another script:

 import cPickle as pickle  
 data = "";  
 with open('genovapkdata.fb', 'rb') as fp:  
   data = pickle.load(fp)  
 print "Records: "+str (len (data));  
 countGovi = 0;  
 ....  
 for onerecord in data:  
      if 'message' in onerecord:  
           msg = onerecord['message'];  
           msg = msg.lower();  
           if 'created_time' in onerecord and '2013' in onerecord['created_time']:  
                print 'record not important';       
           elif 'govi' in msg:  
                countGovi = countGovi + 1;  
 ...  

I discard posts with created_time of 2013 again, because this time I'm analysing 1 post at time, and I'm also counting record with a particular word in it "govi"... of course I did other analysis.

It is possible now do some statistics like:
- know who is the main post-writer in the group
- do a top10 of the place where the people train most
- how many are the generated posts
- how many are the generated comments

Monday, 22 December 2014

Why every developer should attend an hackathon?

In the past 3 years I have joined at least 10 hackathons, most of them in my country(Italy).
This is me, explaining my idea in the last hackathon I took part: Techcrunch Disrupt Hackathon (London): http://on.aol.com/video/extendernote-at-techcrunch-disrupt--london-2014-hackathon-518470061

Hackathons are a great way for companies to find new ideas around their product and test it in different situations and contexts.
These events are a mashup of tecnologies and products, mixed together in a very short time: a situation not easy to reproduce differently and for a lot of companies not economically possible.

When the product is an API or a libraries, an hackathon is pure gold; they are a good investment with tiny effort, that helps companies to make blackbox testing and be able to validate: documentation, libraries, api and discovers problems of their product.
During an hackathons for 24-48 hours, hundreds of developers are testing company's product/api/libraries for "free".

It is not always a win for a company, some projects/products are just ignored by developers during an hackathons but at least a good way to promote a product.

Btw, there are a tons of good reasons also for developers to be part of an hackathon:
- Test your skills: during an hackathon you will need to put your best knowledge, trying to build a working prototype in a very short time. Every problems will keep you far from your objectives, forcing you to reduce the features you want. It's challenging.
- Networking:find people to create a collaboration (designers for example), talking with people of other places and discovering their way of thinking, that is  culturally different and then, it is interesting.
- Prices: good prototype, good pitch, good idea and a bit of luck, with those ingredients it is possible to win cool prices.
- Validate your ideas: do you think to have a good idea to show? don't you want to measure how good is it? It's the perfect situation: you validate ideas against strangers, not only friends.
- Learn new tecnologies from others of your team or improve your knowledge: I've tested ionicframework during an H this year, just to see how usable is, in a real situation (= not ready enough)
- Feeling home talk with other developers about geek's things: new products, libraries, technologies and ideas.
- Find a new job Never gone to an H for this, but it happens that some companies are there to seek new employes. They'll come to you asking for an interview: "what are you doing in your life? What's your job? Why don't you come and visit our companies?" and so on.
- Pitching speaking in front of a lot of people and explaining your idea. You need to summarize, make choices, be clear and convince others of your work in a short time. Plus: do it in a foreign language.
- Enjoy the experience: it is a devastating phisically experience but you will be happy at the end anyway

And remember to be prepared before the hackathon:
- read how many documentation you can, about the aim of the hackathon and the sponsors (also searching on stackoverflow problems with api/libraries)
- write down some ideas you will like to build there and be prepared to join others idea
- choose a platform/tecnologies that you will use during the H
- prepare your hardware: tablet,smartphones, smartwatch ...
- prepare your software: libraries, framework, examples, documentation

So if you are now convinced and you want to keep an eye on hackathons near you, check www.challengepost.com


Monday, 11 August 2014

Smart Charger: automatically backup your photos

[This is the second post about Arietta: the first is here]

The project
The idea behind this little project is to have a system that charge my phone and automatically backup the photo of my android phone while it is connected.

First of all, you need to attach an usb female adapter to Arietta and connect an usb (powered) hub to it: Follow this page

In order to get the photos on the android phone from Arietta you need to use MTP protocol and there are 2 ways:
a - install mtp-tools packages from repository
b - activate mtp filesystem

For my project the first solution is easier:  launch apt-get install mtp-tools

The solution b was better, because it allows you to mount the phone filesystem and copy files from it with normal linux commands like cp,rsync,mv.. but it's harder because you need to rebuild the arietta image.

External Usb Storage
The Arietta will copy my photos on an external storage instead of the internal sdcard of arietta because this helps to improve the life of the sdcard.

To automatically mount an external storage in a predefined directory:
- use blkid command to obtain an id that identify you external usb storage for the system
- change /etc/fstab and insert a line to specify where the storage mount point

My external usb storage is a simple pendrive formatted with fat32 with UUID FC4D-37AA.
At the end of /etc/fstab I've added this line to mount it on /mnt everytime is plugged in:

UUID=FC4D-37AA /mnt vfat defaults,umask=007,gid=46 0 0


The Backup Script ...
I've created a script called photobackup.sh that use some commands from mtp-tools package to copy and delete jpg photos from the phone:

#!/bin/bash

#check if external storage is present or abort
checkmount=`mount | grep /mnt | wc -l`
echo $checkmount;
if [ $checkmount -ne "1" ]; then
        echo "Usb External Storage not mounted...exiting";
        exit;
fi

echo "Usb External Storage Found";
mtp-files | grep .jpg | awk '{ print $2 }' > lista.txt
while read F ; do
    #there is enough space on device
    spacecheck=`df -h | grep /mnt | awk {'print $5'} | tr -d '%'`
    if [ $spacecheck -gt "95" ]; then
        echo "Not enough space left on device..exiting";
        exit;
    fi
    mtp-connect --getfile $F /mnt/$F # copy the file
    mtp-connect --delete $F          # delete the file from the phone
done

Explanation:

- it checks if Usb External Storage is mounted or it aborts the operation
- it creates a list of jpg files (supposing these are all photos) lista.txt


... and for every photo:
- it checks if there is enough space left on the Usb External Storage
- it reads the list of photos from lista.txt and it uses mtp-connect to copy file to the /mnt/ directory and it deletes the photo from the phone


..and the UDEV Rule!
In order to call photobackup.sh everytime you plug the phone to the hub you need a udev rule.

Create a script 98-my.rules with this content:
SUBSYSTEM=="usb", ATTR{idVendor}=="22b8", MODE="0666", RUN+="/root/photobackup.sh"

and copy it on /etc/udev/rules.d

Now everytime you attach the android phone to the hub, while the phone is charging, arietta copy your photos in the external pendrive, if present.

In the next article we will see how to add a led to monitor the status of the operation.

Friday, 8 August 2014

How to hack Clash Of Clans: Unlimited Shield [UPDATE]

Introduction
*updates on the bottom
I've played for over a year with Clash of Clans: a very nice mobile games.
Day by day, the game become more and more difficult to play without spending money.
It's the classical in-app purchase free game mechanism: pay to survive.


The hack
During the game you have a limited shield, that can save your village from attacks for others player for a max of 16 hours.
Of course you can extends the shield with money.
When you have no shield, the others people can't attack you if you are online and connected.
My first attempt was to put the display of the phone in a "never sleep" mode and leave the app open but this was not a working solution, clash of clans see that you aren't do nothing and it disconnects.

Later I was thinking about an hardware solution with arduino, that could simulate at least the swipes on the touch screen, something like this hack for another game: https://www.youtube.com/watch?v=SnUH6f_Mv8o but there was an easier solution.

UiTesting with UiAutomator
One of the android's tools is called uiautomator http://developer.android.com/tools/testing/testing_ui.html it is used to test UI on apps, so it is possible to write test/programs that simulate click, swipe and other actions on apps running on the phone.
What I have done, it has been write a simple uiautomator test that:

1 - it presses home
2 - it searches clash of clans in the list of apps
3 - it launches clash of clans
4 - it waits 15 seconds for the app to start
5 - it does swipes movement
6 - it waits some seconds
7 - it repeats from 5

I tested the app on my phone (no root required) and my village was opened for over 10 hours(see update #) without any disconnection and of course, none attack! :)


Update #1: 18 Aug 2014
Yesterday I posted on reddit this post, to understand the reaction of some players and what they are thinking about it.
On reddit I've discovered that after 6 hours of game, Clash Of Clans automatically disconnect the client for 5-10 minutes... but, my hack (I'm lucky) keep on try to reconnect during this period and at the end of this period it is correctly reconnected.

I have also discovered there are some hacks out there that works with rooted phone and works in similar way.

Friday, 25 July 2014

Experimenting with Arietta: little & cheap embedded linux board

I bought this board months ago when there was a preorder offer on www.acmesystems.it: just 10 euro!
It's a 400 Mhz board, 256 mb ram, with usb support that runs a reduced sized version of Debian Wheezy: Specs Here.



Setup
First of all you need to:
- prepare a micro sd-card: follow this or buy one ready
- plug arietta with a microusb cable to your windows/linux computer (no mac support ..but of course you can use a virtual machine )
- configure Network on your computer: follow this

When arietta is connected to your PC, it is recognized as a RNDIS/Ethernet Gadget.
On my Linux Xubuntu PC launching ifconfig from a terminal I have a new network interface device "usb0".
After the setup you can use "ssh": ssh root@192.168.10.10  with username root and password acmesystems and you have a shell on arietta!

Install Packages
Install new packages on arietta is simply apt-get!

To let Arietta reach internet you can:
1 - connect an usb to ethernet adapter to Arietta, and configure it
2 - use your PC as a gateway to forward the traffic of Arietta through the USB connection

For step 2 this is what I did on a xubuntu machine:
- change sysctl.conf and uncomment from the line to forward packages "net.ipv4.ip_forward=1"
- launch the following commands: the eth1 is the NIC connected to internet

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -A FORWARD -i eth1 -o usb0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i usb0 -o eth1 -j ACCEPT

(Of course everytime you reboot the machine the iptables rules are lost.. unless you change iptables setup)

Now you can connect again to arietta with ssh:
- update repository /etc/apt/sources.list
- apt-get update
- apt-cache search
- apt-get install

Topic of the next article: arietta, libmtp, udev, photo backup... and "juice jacking"


Wednesday, 5 March 2014

Showing my heart rate while playing a scary game: Slender Game


 Yesterday, I've done this video:

Watch fullscreen

It shows me and my heart rate while playing Slender (www.slendergame.com).
With arduino, pulsesensor.com and some scripting I was able to monitor my heart rate.

Normally when I'm calm, my heart rate is around 50 BPM.
If you want to watch the most funny part of the video, go to minute 14 and watch how the BPM changes.

It's not hard to reproduce something like that.

Friday, 21 February 2014

Summary of the last months

These are some of the topics where I put some efforts in the last 3 months:

Leap Motion
I've played with Leap Motion and their javascript library...producing a sort of Flappy Bird Clones that use leap motion as Controller: http://flappyleap.kissr.com/ it's just javascript and it's pretty simple how it works. Just check the soure code. You need a leap motion to play

Unity 3D
I've started learning Unity 3D: it's a great tool to create 2D and 3D games.... It' pretty easy and very powerful: you can create a game that runs in the most popular desktop and mobile platform. My impression is that it is a tool thought more for 3D modellers but good for anyone: I spent some night trying to move my objects in the Scene in a particular way.. thinking about the math...than I found a function that it did just what I was searching for. I've created a 3D game called Apocalipto for BlackBerry10 http://appworld.blackberry.com/webstore/content/42671888/ using just the free assets on the Unity store. It was started as "Zombie and Shotgun" but there is no free 3D zombie in the assets store, so they become Skeletons... than it was hard to bind a shotgun correctly to my main character and it become just Apocalipto ... :D

an old version of the game.

This game support bluetooth joypad connected to the BlackBerry smartphone. BlackBerry sent me 2 really good joypads just for adding the Gamepad support to this game: http://devblog.blackberry.com/2013/11/up-up-down-down-left-right-left-right-b-a-gamepad-offer-update/
Think what you want... but the BlackBerry efforts and support to developers is really great.
I think I will publish this game also on play store... I hope soon.

Bluetooth LE and the iBeacon world
It's a pretty interesting topic.. 
Bluetooth Smart will be a huge adopted tecnology in the Internet of Things world(another hot topic): so I've started learning and experimenting this tecnology in multiple platform: OSX, IOS, BlackBerry and Android. 
I've bought and played with different device: bluetooth 4.0 dongle attached on linux, the Texas Sensor Tag and the Coin (an arduino mini pro with a bluetooth 4.0 chip attacheda also able to simulate an ibeacon/estimote).
I spend a lot of time testing IBeacons: you can use a product like estimote, or you can use just a simple bluetooth 4.0 usb dongle with Bluez and some tools.

Google App Engine
Another thing where I spent some of my late night time.
I think GAE is an incredible backend for services or apps.. it is easy, powerful and easy to scale.
But the first reason why, I've choose to learn it, it was Google Glass.
Google Glass can use apps called glassware, part of these apps are software that talk with glass through google cloud.
So, I've started to learn it, just to be ready for the time when I'll create a glassware.
I've created a couple of services with python just to understand how to store data, add authentication and show pages to user.
Anyway, it's the kind of service you can rely on, if you create a great app.


Chrome Web Apps
I've built a simple quiz/game for a contest on opendata: http://pro.europeana.eu
The game is called "Europeana: guess who?":
https://chrome.google.com/webstore/detail/europeana-guess-who/jameiloaoadpfonmpmhkmahpdnmnnaka
Nothing revolutionary...
The interesting things are:
- you can use html and javascript (with some restriction) to create your extension
- the extension easily become desktop application and with this, I mean, it can be launched from your windows or osx launcher (I didn't try it on linux) and they appear like a normal application.
- transform a chrome extension into an android and ios app (not ready for ios right now): http://blog.chromium.org/2014/01/run-chrome-apps-on-mobile-using-apache.html it uses cordova. I've tested the tool to transform my extension into an android app, it works but you have to built an extension with a design that runs well on very small display(phones) but also on very large display(desktop).