Talking with Thread

With another search on the web.. I have found the answer I was searching on yesterday: Queue with thread.

How? you can read here a good guide.
So anytime the code need to fetch data from the web, there is a good thread who can manage it.

I would like to see a urllib2 library completely asynchronous.. but for now.. you need threads.

Other things I did..a lot of bugfixing and a little bit of code cleaning.

I have also found another bug while displaying data on the chart.
Next step, will be chart improvements... and than..

first alpha will be out!

Baby, did a bad, bad thing

Yesterday I wasted a little bit of time searching to find a bug: I had to delete a selected element from the list of registered stocks in the main screen, and also from the internal db.
Everytime I searched to delete one element from the list, the deletion of that element from internal db doesn't work... instead it was deleting another element.
The code was something like that:

1. get the iterator and model , for the selected element of the treeview.
2. remove the element from the model giving the iterator.
3. the iterator became invalid...(yes it is deleted!) so I did the same thing of step 1.
4. remove the element form the db getting the key from the iterator.

The bug was simple but it takes some time.. at point 4, the selection to the treeview changed to the previous element in the list!
So the code changed like this:

1. get the iterator and model , for the selected element of the treeview.
   
2. remove the element form the db getting the key from the iterator.
3. remove the element from the model giving the iterator.

Now add and delete of stocks works.

The other problem is with pygtk and thread.. As I mentioned before.. everytime the application try to connect to the web the interface freeze until this operation is complete.
This happened because retrieving data from the net, it is a blocking operation, and block the main thread where the GTK engine runs.. so the application freeze.

I read so many pages about which is the best way to implements a Thread.. and I hope to be near to a solution.. but not yet!

Add functionality & autocomplete

I have added a textentry on the top of the dialog for add a new stock quote.
This textentry helps to find through the list of available company, with an autocomplete algorithm... while you type, the list change.
The Add, now it works!

In the main screen when you select one of the registered stock quote, the chart updates, but I have some problem displaying the chart..
I need to examine more later on this, I have a pycha error with some company and I have to understand if it is a problem with the loaded data or a pycha problem.

Latency
Every network operation is blocking the main thread.. so the UI freeze until the dowload of data from the yahoo finance complete: when you add a new stock, at startup and when chart updates.
A disaster! :D

This is a common UI problem and the solution, here, it is to use a secondary thread that manage the connection.
I have to search for which is the common solution on pygtk.. maybe there is a good way to do this, or a pattern to follow.

It is growing day by day!

I think the final name of the app will be UStockQuotes!

This is a screenshot of the last evolution of my app:

As you can see there is a list of stock quotes now.
This is the list of the stocks interested by the user.
At the moment the data is stored inside a database, the application fetch the list from the database and than fetch the updated data from yahoo service.

When you select one of the row of the table, the chart update itself retrieving the data.
 
So this is roadmap, just for the first part:
- add/remove stock quote.
- handle no-selected quote, no-stored-data (leaving a banner for example or a message), yahoo errors: means no plot.
- change chart period.
- change stock quote order in table.
- data caching mechanism for fast response and less requests.
- loading screen.
- better unity integration.
- ppa & preparation for contest.


Nice to have:
- Fullscreen mode.
- Fullscreen with stock quote data-rotation.

Main Screen Mockup

 I did some test with Yahoo Finance Api.. trying to understand, which data, it is possible to retrieve fast from their service and which is slowly, also because it is huge.
What I would like to achieve in this application is  a fast, responsive interface, designed to mitigate the delay during data retrieval.
A good organized application can help to reduce this problem and make users happy to use the application.

After this step, I did a mockup, with pencil, of the main screen of my application:

Let me show you how it should work... using the 3 bullet on the mockup

• bullet 1: this is the zone where chart of a selected company is displayed. At the first launch will display the chart of the first present company or, a message if none company is registered. As you can see, there is also on the top right a combo box. This combo box contains some prefixed , period: 1 week, 1 day, 1 month, 1 year ecc. Changing the combobox will update the chart accordingly to the selected period. I don't know if I will use a combo box, but it's just for the idea.  I would like also to change the period, simply scrolling in or scrolling out on the chart.
• bullet 2: The list of your selected stock quote companies. At the launch of the application this table should be populated asynchronously .. putting every line , one by one, after every download of the information from a company, and making user informed of the in progress operation. Clicking on one of this row will update the chart.
• bullet 3: a toolbar for handling: add of a new company in the table, remove of an existing company, fullscreenmode. Fullscreen will be a mode to use all the screen for displaying the data of only one selected company with a chart that will take 90% of the screen. It needs a new mockup only for this.

I will try to reduce call to yahoo service whenever is possible with a local caching mechanism like save the data displayed on the table, save the data displayed on a chart for a company... So if I know that the data is locally present, I will not fetch it from Yahoo, otherwise I will fetch, display and save it locally.

Other Features..
Yes, it is not just finished here but, my time is limited so I need to proceed step by step and reach some milestones before proceed with other feature..
I prefer to create a working program with less feature but robust than an application with many feature and bugs.

About Dialogs and Gtk.TreeView

Just finished working on this: open a new dialog with a simple treeview inside and other widgets.
This dialog will be used for searching inside the stock quote table(see the previous post)

First problem first
Adding a new dialog to your project it is easier than you think.. if you know that you should use Quickly (and if your project is quickly based)!

Quickly team make easy to add a new dialog to your project.. just follow this answer.
I wasted an hour searching for an answer on how to manage it with glade3 and finally I found that post.
I like this solution... it is easy to add a new dialog and it manage files for you.. just quickly.

After a dialog finish his task, it return a response code; a list of response code is here.

Second problem.. Treeview, pygtk and Glade
This is a complicated task... information on the net are poor or refers to old version.. but you can do a lot with glade3.
You need to create a TreeView and the associated ListStore.
On the treeview you should define:

• column list with information
• the renderer for every column you need like Text, Image or other widget available

On the liststore you have to define:

•  the column to use with the type for every column (for text gchararray is enough)
• eventually, any static information you need in the list when the treeview start.

ListStore need to be connected with Treestore and also you need to map every cellrenderer to an attribute on the liststore! For this last point, there is a "Text" property in the tab General of a CellRenderer element.
If you don't map the attribute and you have elements on the liststore, you will probably see 0 lenght string on the treeview.

What now?
Now, I have enough informations to stop hacking and start designing the app.
Also I will try to include the feautres that someone asked on reddit :).
Need to sleep now, it is a working day tomorrow and it's too late!

It's Symbol not Keyword

Thanks to this reply from Stephen Lepage I discover that the correct word for searching company in stock quotes data is Symbol and not Keyword!
So GOOG, AAPL and RHT are the symbols for Google, Apple and Red Hat!

If you want to check data from yahoo finance, you have to pass this data inside the request.

Where can you find all the symbols or, at least, a good number of it?
After a little search I found this post on stackoverflow (the saint of all developers) where the first answer said:

you can also get nice tidy csv files from nasdaq.com here: http://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=nasdaq&render=download (replace exchange=nasdaq with exchange=nyse for nyse symbols).

Those links took to a downloadable .csv file that contains the Symbol, Naming and other information of all the companies form Nasdaq and Nyse.

So the next step for my application is to make this data easy to query...yes.. query.. so.. I made a python script that:

• read data from csv.
• read only the necessary information.
• save the new data to a sqlite3 db.

And this is a search on the table:

This table will be used in the application for lookup operation of symbols.
The db, will not be used only for this operation but for other stuff.

The application is also on reddit now: http://www.reddit.com/r/ubuntuappshowdown/comments/vge3x/app_submission_a_stock_quote_app/

Data downloading... complete!

Ok here it is again.
Today, I'm happier than yesterday.. just because the things went as expected and I can go to sleep earlier...

This is how it looks the new experiment:

That chart display google stock quote (on a predefined period) integrated with pycha library and gtk3!

What I learnt
If you want to:

• draw on a GTK3 drawing area, you need to register to "draw" event.The "expose" event doesn't exist anymore.
• update a drawing area widget, you must call queue_draw on that widget
• get the dimension and position of a widget, you can call the methoed "get_allocation()" on that widget. This methoed give back a Rectangle object with this properties: x,y,width,height.
• plot with pycha, you have to create a context from the drawing area widget and from the context a surface, than pass the surface to pycha. More info on this point soon.

Yahoo Finance and Stock Quote Data
The stock quote data is obtained from Yahoo Finance, thanks to a web services present on their portal.
Thanks to Corey Goldberg with his library ystockquote obtain these data is a pretty easy task.
I have only to understand how much deep I can go with this script and if it is enough for what I'm looking for...
I also don't know where I can find the right keyword for stock quote companies: Google is GOOG , Apple is AAPL but I don't know where to find a list of these keywords.. or better a webservices to query.

After this research.. I need to pass to design & a bit of mockups!
 

Lack of documentation

Yesterday I spent a lot of time trying to plot a Chart in a Gtk Drawing Area, using quickly, Glade3, pygtk and pycha.

Some criticism about these tools:

• "hackish" style... you fix some problem thanks god to people like this: http://somethingididnotknow.wordpress.com/2012/01/27/retrieve-the-window-object-from-widgets-with-pygobject-gtk3.
• general lack of documentation. what you find around the web is usually deprecated and it doesn't work when you try it
• the gtk documentation is mainly in C, not in python, and sometimes have a C function called pippo() doesn't mean it exists the equivalent python function pippo()..but something like pyPippo()....
• when you need a library for a simple task, like plotting on a screen, you can find a lot of solutions, but find the one that just works , it is hard. Many library examples just doesn't work.. you have to find the solution. Read this http://detrasproject.wordpress.com/2011/04/04/show-graphs-using-pygtk/ Also if you don't want plot chart, read the post to understand what I mean. It's not coding.. it's hacking.
• Trying to plot a line chart inside a drawing area...it has been a bad experience.. and the result not satisfying for now... it looks good but there are a lot of bug to fix.

 

a prototype.

Quickly, pygtk, glade etc are great tool.. but Ubuntu need documentation, keeping it updated with examples and easy to find tutorial on "how to solve a simple task" or a page with the most important and working library for solve a task.
The hackish approach is not for all, we are in 2012.

This is not an attack to ubuntu community or in general to opensource.
Instead it is an important point, where community should think and improve.

I will post later the recipe for plot a pycha inside a GTK window made with Glade.

Planning and some code

For many reasons, the only way I had to find a place for ubuntu 12.04 was some space on Virtualbox.
Ubuntu on Virtualbox is quite fast to don't regret a real machine.

I have seen some parts of the video recorded from UbuntuDevelopers on this youtube channel: http://www.youtube.com/user/ubuntudevelopers
If you are new to pygtk, glade and quickly I suggest to follow those videos.

Planning
From now on, I have around 15 days to produce something good for the competition..
The 15 days will be splitted like this:

• 3 days experimenting, searching and understand new library in python: fetch webservices,  plot graphs (at least lines) and tools pygtk/glade, quickly.
• 4 days for design
• 4 days for programming
• 1 day for testing
• 1 day for submission
• 2 days extra time.. I know those days will be used in some way.

1st Problem
Yes, 1st problem: I desperately need to find a way to plot "cool" graphs with python.

I searched a bit and what I've found is not satisfying me enough.

I don't have the time to create a new library for plot graphs like I want..

Maybe, at the beginning I will use matplotlib (see this tutorial http://www.thetechrepo.com/main-articles/465-how-to-create-a-graph-in-python ) and I will try to improve how graphs looks on the screen only if I will have the time to do it..
I hope matplotlib is able to draw directly on a Gtk Drawing Area... I don't want to save on disk the image than load an image on the Gtk Drawing Area..... it is a sad approach.

A Stock Quotes App for Ubuntu

I will do a simple stock quotes application with Quickly for Ubuntu 12.04 but first of all..

Why did I choose to make a stock quotes app for Ubuntu?
Everything started some day ago, while I was playing with the new beta of IOS 6, I found the IOS default app for trace stock quotes information:

It was the first time that I ran that app.

I like the way it works:

• built to be easy
• fast manipulation of information: add/remove stocks quote
• fast searching for stock quotes
• interface without noise
• smooth animation and graphics

I've never been interested in stock quotes, markets, etc.. but when you find an application like this and you feel how it is designed...you are captured by how it guides you.

After that I see there is an Ubuntu competition and I have thinked to make something similar for Ubuntu.
I am an Ubuntu user since 4.10 and I would like to give some of my contribute back to the community (I did in other way in the past but I never did a program for the community).
I see there are other stock quote application but I don't like it.. and I have some idea on how to create something different... I would like to create a "wow" effect app.. maybe
I will use quickly (as suggest in the competition) and I will use this blog to document my progress.
I hope the future informations posted on this blog will be useful for others.

First!

Who Am I? 
Hello everyone,  I'm a 28 years old  developer, from Genova (gmaps) a nice italian city near the sea.
One of the typical plate here is Trofie al pesto (that's why pesto in the name).
When I was a child, I usually break toys trying to change it, transform it or simply trying to make it works again... in a simple word hacks  (yes, this complete the name of this blog).

My fist computer was a fantastic sinclair zx spectrum 48k, my love with computer and technology started with that cute gummy keyboad at the age of 6.
From that moment, obviusly, things changed a lot..and as any good kid I spent a lot of time learning from computer!

In many years I have learnt many languages and technologies like: C, C++, python, java, php , bash , ios, android, j2me, gtk programming, javascript, html, css, linux, xml, xslt, sqlite , oracle, cloenzilla , windows, osx and many others stuff..
Everytime I found some tech-thing that catch enough my attention I start to study it...
And of course I do a lot of experiments with tech-toys like this  http://extenderconcept.wikidot.com/
My computers are only mac and linux workstation, it is easier for me to be productive with those OS.

 
Why did I start this blog?
So.. this is more important for you reader..
I decided to start this blog because:

• This contest : http://developer.ubuntu.com/showdown great idea Canonical! I use it as an excuse to start this blog, so I have a trace to follow for my future posts about my application idea ( more info soon )
• I'm italian..my english is not so good and I hope to improve it writing here.
• I hope that the stuff I will produce will be useful for others and to receive feedback. I love to improve my knowledge.
• I like to share ideas with others: be a linux user from 10 years means something about sharing.