Always learning

Ubuntu adventures - django install a breeze!

2009-04-21T14:55:00.000-07:00

Installed django on ubuntu today. Do not make the mistake of installing django from the synaptic package manager. Firstly, its an older version than the current 1.0.2 and secondly, it does not install it in your python "site-packages" folder.

The best way to do it is as they say on the django site. Download the tar file and install using python setup.py install. The cool thing about this is is that I found that the symlinks were also created for you. I could access django-admin.py from any directory in my machine.

I also installed postgreSQL and psycopg2 (the python interface for postgres) from the synaptic package manager and I was good to go with django in a matter of minutes! For now I am developing with the webserver bundled inside django but I hope to be switching to apache with mod_wsgi soon.

I hope to be tinkering a lot with django over the next few weeks.

Perl power! Sorting dictionaries (or hashes in Perl lingo)

2009-04-21T14:39:00.000-07:00

Theres a dictionary dict1 with keys key1, key2, etc such that $dict1{keyn} has a value which itself is a dictionary. Lets call this dictionary %subdict. Now subdict has different keys subkey1, subkey2 and so on such that $subdict{subkey5} = subvalue5.

Problem: How do I sort dict1 such that the keys are in the ascending order of subvalue5?

This would not be easy to do in a single line of code, unless of course the language is Perl! Ready? here it is:

dict1_keys_sorted = sort {$dict1{$a}{subkey5} cmp $dict1{$b}{subkey5}} keys %dict1

Lets take this apart. Going from the right side, you start of with dict1. Now, keys %dict1 returns a list of the keys. With the sort command being invoked, Perl feeds the first two values of the keys into $a and $b. So, $dict1{$a}{subkey5} gives the value of subvalue5 for that key value of dict1. The values are compared and depending on true or false value, it puts the value into the new list dict1_keys_sorted which is sorted.

Note: "cmp" is for sorting strings. To sort number use "<=>".

Ubuntu journey II - Printing and scanning

2008-05-13T09:03:00.000-07:00

Printing and scanning in Ubuntu proved to be much easier than expected. Ubuntu easily recognized the HP Photosmart C3500 series that I had, and right away I was able to print documents off the text editor. The Scanning application (XSANE) was a breeze to operate and I was able to scan documents to various formats.

However, my favorite lightweight editor on windows was SciTe, so I installed it in ubuntu also, and tried to print from it. Turns out SciTe uses the "a2ps" program to print, and it was not able to direct the output to the printer. A bit of googling told me that replacing a2ps with "enscript" would be able to do it. So I replaced the print options in SciTeGlobal.properties file, but now I get an error that its not able to find the printer class. Guess I will have to look into this more.

But besides that, the printing and scanning functionality works like a charm in Ubuntu.

Ubuntu journey - Hardy Heron

2008-05-12T09:35:00.000-07:00

I just upgraded my ubuntu distro to Hardy Heron (8.0.4) from Gutsy Gibbon (7.10). I did not use it a lot when I had Gutsy but now, hearing about how much better the distros are getting everytime theres an upgrade, I am setting out to explore it more. The major difference I noticed in Hardy was that I am able to make ubuntu work on dual monitors directly without running additional scripts. In Gutsy, I actually had to run an XRANDR command everytime I logged in, so that I could switch it from clone mode to dual monitor mode. Not anymore in Hardy.

Further things I will be exploring on Ubuntu in the coming days:
- Printing and Scanning
- IPOD and music play
- Python, Django
- Java Development with Eclipse
- Ant, CVS, Tomcat
- MySQL

If I find that the above are easily setup and software development in that environment is as easy as windows, theres no reason why I should not completely switch to Ubuntu.

Oh I will have to go back to windows frequently though, since Cricket games for PC do not run on Linux yet!

Thoughts on Agile techniques - I

2007-11-28T14:06:00.000-08:00

Over the past few years, I have observed the agile movement and I feel that I have to put down some of my thoughts regarding this. The main reason for me writing it is to clarify my thoughts, and also, but not necessarily, to invite some feedback.
I cannot dispute the fact that in the process of evangelizing it, the people have talked about some software practices that contribute to making better software. But somehow I feel that the useful practices are all those that have to be part of writing software in general, rather than being attributed to a process.
Its my belief that there are mostly only two qualities that a developer has to have in order to deliver successful software. This is of course only in context on enterprise software development where the developer in employed to create and maintain internal software systems within the company. The two qualities are: one - not being afraid to go in and change code and two - constant communication of the status of his work.
The second one is an auxillary capability, since it solely exists to put people mind's at rest, so that they don't bother him/her while he's actually trying to solve a problem. I would think its an equally necessary quality to possess as the primary one because when it comes to deploying a fix or a system, its never only about the code. There's always a dynamic between the current users that have to look for a workaround or need to be informed about systems that will be down, etc.
In such an environment, if the developer possesses the confidence of being able to tackle changing the codebase or developing something new with the confidence that it can always be changed later, there's not many more practices that he needs to religiously follow to deliver working software.

This confidence comes from constantly learning from a good amount of experience as well as possessing a couple of tools that come in handy. I will delve on this aspect more in the next post.

The beauty of Python Generators

2007-03-06T13:14:00.000-08:00

Say you had to read a file line by line and change the delimiter from comma to semi-colon. The python code would look like this:

f = open("c:\\testfile.txt") 
for line in f.readlines(): 
    ';'.join(line.split(','))

Now say you wanted to do the same thing, but only to the first 5 lines of the file. The impulse for a Python newbie would be to write the following code:

f = open("c:\\testfile.txt") 
counter = 0 
for line in f.readlines(): 
  if counter <> 5 
    ';'.join(line.split(',')) 
    counter += 1

A more savvy python newbie would write the following code taking the advantage of the beautiful enumerate() function:

f = open("c:\\testfile.txt") 
for counter,line in enumerate(f.readlines()): 
  if counter <> 5 
    ';'.join(line.split(','))

The problem with is code is that it has two concepts intermingled, one to read 5 lines and the other for the actual process of changing delimiters. If we wanted to keep the concepts separate, we would have to be able to write something like this:

f = open("c:\\testfile.txt") 
for line in f.readfirst5lines(): 
  ';'.join(line.split(','))

Here the process of stopping after 5 lines is encapsulated by the readfirst5lines() method in the file object and the method body only does the changing of delimiters as before. Now this code will actually work! The reason that we can write code like this is due the python generators feature. readfirst5lines will look like this:

import itertools 
class myfile(file): 
  def readfirst5lines(self): 
    for i in itertools.count(): 
      if i == 5: 
        break 
      next = self.readline() 
      i += 1 
      yield next

This is of course more lines of code, but the concept is abstracted away nicely. We are separating the conditions for processing the file from the actual processing. And whats more, this method can be slightly changed to take number of lines as the argument. So if you want choose to read the first 2, 3, 5 or how many ever lines, the method will look like this:

import itertools 
class myfile(file): 
  def readfirstfewlines(self, n): 
    for i in itertools.count(): 
      if i == n: 
        break 
      next = self.readline() 
      i += 1 
      yield next

And you can use it like this:

f = myfile("c:\\testfile.txt") 
for line in f.readfirstfewlines(3): 
  ';'.join(line.split(','))

This will process the first 3 lines.

(Note that we used the 'myfile' constructor method instead of the 'open' function to open the file, since we need an object of type 'myfile' and not 'file'. There are other ways to downcast in python but this is probably the simplest way to do it in this case.)

Python generators go a long way in making the code more elegant and encapsulating separate concepts. I will blog about the mechanics of how generators work and its other uses as and when I learn more about them. For Rubyists, this concept of generators is pretty much similar to "blocks".

[Rails] How to work with non-autogenerated primary keys

2006-06-30T09:36:00.000-07:00

Oddly enough, I have not found a clear explanation of how to make ActiveRecord work with a table that has a non-autogenerated primary key. In other words, the table has a column as a primary key for which the value has to be inserted manually. (Of course this is not good practice but what other better reason to do this than to say such a table already exists and needed a user interface for maintaining its values!)
So how do we go about doing this? Simple. First, set the primary key in the Model object to be the column in the table (since ActiveRecord always assumes primary key is a column called "id"). If we just run this and try to save the model object by firing up the rails console, rails gives an error saying there are no attributes "col1". This is because, rails assumes that the primary key is not editable. So all we have to do is define attributes for the primary key column col1. The overall changes are shown below:

class MyModel < ActiveRecord::Base

set_primary_key "col1"
def col1
read_attribute "col1"
end
def col1=(value)
write_attribute "col1", value
end

end

Now fire up the rails console and create and save the model object, setting values for the required columns. Voila!

Note: since you are departing a bit from rails convention of primary keys, you will need to make sure that the required validations are added in the model so that you dont get errors from the database.

Data comparison? Python file comparison to the rescue!

2006-05-31T10:06:00.000-07:00

I had to update some data in a production database for a new functionality. I had already done that in the test database during development, but it was not just a simple insert. The steps that I did went something like this:

1. Generate a file with a set of insert statements (used Python for this!).
2. Manually change some data in the insert statements.
3. Run the insert statements

Now since I had to do the same in production, I wanted to make sure that after this insert, the data would look exactly the same as it did now in test. In other words, I wanted to be able to repeatedly run the process above, and make sure that the steps produced always the same set of data. This would guard against any mistakes being done during step 2, which is a manual edit of a file and prone to error.
Sounds simple - I could generate the file of insert statements, and then compare it against the file that was already changed and used to load the data in the table. Problem is, I no longer had that file I could compare against. So instead of comparing insert statements, I would have to compare data. The process I did went something like this:

1. Export current data in the table to a file, say file1.txt
2. Delete the inserted rows from the table.
3. Generate insert statements again, manually edit, and insert rows into the table.
4. Export data in the table to a file, say file2.txt

Now all I had to do, was compare file1.txt and file2.txt. If they are equivalent, it means my manual edits were on target.

So how did I compare the two files? Python to the rescue! This is all it took:

import filecmp
filecmp.cmp('file1.txt', 'file2.txt')

If they are equal it returns true, else false.

I have to admit that that the only reason I thought about comparing data rather than the insert statements was because I was sure Python would have something neat for file comparison. The database-oriented guys might have fired up some magic by using stored procedures and temporary tables, and it would have been equally satisfying to them.

Its all a matter of what you are comfortable with, I guess.

Churning out Rails dishes - I

2006-05-30T10:44:00.000-07:00

What I learnt from Rails Recipes today:

You can run tests in rails without connecting to the database
test_help.rb is reponsible for the database connection code, so remove reference of test_help.rb from test_helper.rb put in the code yourself. Remove the lines that access Activerecord.

You can connect to multiple databases
All model classes inherit ActiveRecord::Base's connection. So create a parent model class that inherits from ActiveRecord::Base and override the establish_connection method to access the other database. All models inherited from this model class will connect to the new database. Make sure not to instantiate the parent model class since theres no corresponding table.

Integrate with legacy databases
Use attributes and methods like ActiveRecord::Base.table_name_prefix, ActiveRecord::Base.primary_key_prefix_type = :table_name_with_underscore, ActiveRecord::Base.set_primary_key, ActiveRecord::Base.pluralize_table_names to tailor your model class according to the table specifications.

Avoid code repetition(use DRY) in your database configuration YAML files
You can define a section and give it a key. Using that key in various places in your yaml file will substitute the values from that section in those places.

Preparing for RailsConf

2006-05-26T08:32:00.000-07:00

When you go to a conference, its always a better experience if you are somewhat knowledgeable about the topics. :) Otherwise the only benefit would be the t-shirts and coffee mugs! And even though the price of RailsConf is a steal judging by the speakers and the topic itself, its quite a bit for just conference paraphernalia!
So starting today(yesterday actually), I am going to be reading up on Rails Recipes, a wonderful book by ChadFowler et al, that details tips and tricks that you an do with Rails. The plan is to read or work through 3 or 4 rails recipes a day. Hopefully this will get me into "Rails mode" and I will be able to avoid a blank, stunned stare when the magicians whip up their rails magic!

Onward to Railsconf 2006

2006-05-26T08:00:00.000-07:00

This is the first conference I am attending in quite a while. Last one was XML Devcon 2000! It was good, but not as enriching as I hoped it would be. I suspect that maybe because of 2 factors: one, the general concept of XML was not very appealing to me, ie, while developing, consuming or writing out xml, it does not give me a rush. Second, my understanding of software concepts was clouded too much by what was happening in the world, without me having my own perspective. As folks dished out their own takes and the "one right way" to do stuff, I lapped it all up, not thinking much for myself.
Six years later, I dont how much has changed on my side, but I do like where I am now. I do have perspectives on what "superior" languages or frameworks are, and I do feel a certain rush when I understand or implement concepts. At the same time, having these perspectives has also oddly left me with more tolerance for accepting other concepts or approaches that were previously frowned upon by the "best practice" advocates. So with this bent of mind, I am eager to see how my experience will be at the conference.
Besides the fact that the conference is about a subject that I love developing with, it also boasts speakers that I have always wanted to hear - like Martin Fowler and Paul Graham. In the beginning of the year, while announcing the keynote speakers for the conference, Chad Fowler said "If you can only make it to one conference this year, let RailsConf be the one".

I am happy I listened.

Debugging python with IDLE

2005-11-22T13:43:00.000-08:00

If you are playing with Python and eager to get the feel of this wonderful language, you might be already using the interpreter and the in-built GUI "IDLE". I have been using the interpreter to explore the language's various commands and features and its been very easy to use. However, I tried to use the debugging feature of the IDLE tool a few days ago and struggled with it. To me it seemed like someone who has a background of using debuggers in PowerBuilder, Eclipse etc, would find it rather hard to grasp. So if you think you dont know your way around the IDLE bugger, here are the steps. Once you follow these, it will actually be quite easy to debug your favorite Python script. The hard part is getting used to it, but I am sure we can suffer through a little discomfort - just for the sake of Python!

1. Start up IDLE and as soon as the shell launches, click on 'Debug' menu and then 'Debugger'. This will open up the Debug popup. Do not close this throughout the session.
2. Make sure to check all 4 checkboxes, including 'Source' and 'Globals' (Why this is necessary is explained further below)
3. Open up the file you want to debug using 'File/Open'.
4. Right-click on the line you want to set a breakpoint on and click 'Set Breakpoint'.
5. On the menu of the window on which you have opened the file, click 'Run' and then 'Run module'.

This should start the debug process.

This debug feature also debugs any code you enter through the interpreter! So if you start the debugger as outlined above in steps 1 and 2, and then type commands on the interpreter, it will open up the relevant file and walk you through the source. Cool huh??

A couple of glitches: If you dont check the 'Source' checkbox on the debug popup, you will not be able to see the interpreter going through each line of the code. Also, even if you check the 'Source' checkbox, you will be able to see the lines being highlighted only if you take the focus away from the debug popup and set the focus on the window with the source file (say by clicking on the titlebar of the window)

Other than that, the IDLE debugger is very handy in giving you a picture of how the code progresses. As any good developer would say, debugging should NOT be the only way to explore a program but its a good thing to switch back to sometimes!

What makes a good IT manager?

2005-11-15T13:52:00.000-08:00

Sometime back while commenting on a blog entry by Ravi, I managed to elucidate what my idea was of a good IT manager.

1. Someone who makes every team member comfortable in his presence. This is actually very difficult to achieve and has to be done only by letting one's guard down, admitting there are areas which the manager doesn't himself understand, and is very eager to go a developer for help.
2. If the developer is having a technical problem the manager says "hmm cmon lets see how this will work" and actually sits down, pairs with him to do a tracer bullet - a small proof of concept.
3. The manager should show by his actions that he is concerned by the well-being of the team. This means hes genuinely interested in the developers career growth and to some extent personal issues.
4. Last but most important, he sits with the team and all the team members sit together. I am a very strong proponent of this and really cringe when I see people isolated in cubes.

This might be in conflict with some people that think managing is removed from the technical aspect of what the manager manages, but I believe that the opinion is flawed. In short, if the manager has not had adequate technical history, he will always be found wanting regardless of how good he is at dealing with people. This of course assumes that the role of the manager is to make sure that the product or service developed by the "workers" is delivered on time to the customers and reasonably within budget.

Convention over configuration...but of course!

2005-10-03T08:36:00.000-07:00

I have been reading and working out examples in the book Agile Web Development with Rails, written by Dave Thomas and David Heinemeier Hansson. I discovered rails about 6 months ago and started reading about it. I was fascinated by the simplicity and the ease of development that it offered in quickly putting together a fully functional web application. Needless to say, in just over a year after its creation, it has captured the hearts and minds of programmers and web designers alike.

There are many facets to rails that make it worthy of being the one of the foremost choices in developing web applications, like clear separation of model view controller functionality and in-built testing support - besides the inherent advantage of having a dynamic language like Ruby at its core. However, there is one more rarely-exploited paradigm that it uses that really struck a chord with me - the fact that it stresses "convention over configuration". Rails does not have XML files for specifying the configuration aspects of the application. It assumes certain values and uses those to power the rest of the application.

What does this exactly mean? It means that configuration information is already predetermined by rails and it urges the programmer to name objects a certain way and put files and resources in specific predetermined directories. One would think that this is very restrictive, but the fact is that it doesnt seem restrictive at all when most of it is done by the framework itself! It creates the directories, creates the templates for the files, so that what falls on the shoulders of the programmer is the actual business logic. It is like it says, "Dont worry about maintaining configuration info, I will lay it out and make sure its all in place. You just worry about implementing the requirements in code."
Imagine what this means in terms of application maintenance. A new programmer entrusted with making changes to a rails application coded by another developer already knows where everything is, and how the basic components are named. No flailing around trying to make sense of variable and object names!

It definitely is worthwhile trying out and working on a framework that works on the "convention over configuration" principle. This may not be for all, but is perfect for a framework if it has to be true to its promise of blazing speed in putting together a working application. And Rails does a great job there, no question.

The Craft of Classification

2005-09-13T14:14:00.000-07:00

Ravi mohan writes another thought-provoking post on this blog in response to the notion of classifying programmers into 3 categories - apprentice, journeyman and master.

I have to admit I was a bit swayed by Pete mcBreen's book when I first read it - mainly because I had read about that idea for the first time. But then Ravi is onto something when he shows how that classification may have no real value. The book talks about a situation where it says its okay for a "master" to learn a new language from an "apprentice" and that this doesnt invalidate him of his current category of "master". How is this possible? A student is always supposed to see himself as lower in that relative situation when he learns something from a teacher.
When Pete McBreen and others advocate this idea, I suspect they have an underlying assumption about the quality of the learning. Maybe they are saying that the master already has the fundamental concepts down(which is a higher quality of learning) and he just learns another language syntax which is of a lower quality of learning. But will this always be the case?
Imagine a UNIX guru in the league of Eric Raymond or Richard Stallman. Lets call him Mr X. He has been working on UNIX for years and written dozens of open-source tools, is very active in communities and is usually approached when his UNIX peers have issues. According to McBreen, he would be a master. Lets say he is hired by a shop that does development mostly in UNIX tools but also have a smattering of applications in Scheme. He starts working with a candidate who has a good knowledge of Scheme but is not active in the public domain (as a result of which he is not well-known). Now, I am sure Mr X will definitely not breeze through Scheme, since its not just a new language - it requires a different way of thinking. Now in this situation, there is a lot of "higher" quality learning involved where the "master" learns from the "apprentice". Whos the master here then?

All my thoughts are based on the following premise:
There are too many varied things in software for them to be grouped into a "general" category that somebody can be said to be good at.
If somebody has sound concepts of OO design, he may be quick to pick up Java, C, C++, C#, etc but it may not be easy to pick up Python or Ruby or Smalltalk. That is to say, it will not be easy to think "the Ruby way". And also by the same token, sound programming concepts in even Python or Ruby or any language will not make you a good webdesigner, adept at CSS, HTML or Flash.

I guess then what can conclude is:
1. The only thing one can categorize a person is with respect to a particular paradigm and/or language ("OO programming", "Functional programming" is a paradigm - "Java", even ".NET" is a language for purposes of explaining). If he has to work on another area he doesnt know, he is a novice there and there is nothing wrong with that fact. So Mr X is a UNIX "master" but an "apprentice" in Scheme. There is nothing wrong with a classification which relates to a particular technology/paradigm/language/framework.
2. The only way to categorize a person with respect to a technology is by seeing his work.
There can be no other parameter. No writings, no talks, or drawing on the board. Now, the question is, how do you know if his work is good? Sadly, it cant be judged right away. This can only be decided over a period of time. Say over a course of 2,3,5 years, how many times was it changed? How easy was it to change? Has he constantly delivered running, "not-getting-in-the-way" software to the users?

After reading McBreens book, I started thinking, "I am an apprentice now..I want to be a journeyman in X years, and then a master in a another Y years." Looks like it wont work. I am now going to say - "I know java and I want to learn Python this year. Next year I will dig into Scheme. And hey, OCAML looks interesting, why not try it after that?" And when I get that job in the Python firm that wants me to maintain some apps in Erlang on the side, I will seek out the "Erlang guy" and become his "apprentice"!

This may work...

2005-08-24T13:11:00.000-07:00

I think writing and keeping an online journal does help in quite a few ways. This weblog is to explore, refine, clarify and elucidate ideas in software development - a profession that has helped me earn a living, and by virtue of doing which continuously for a few years, I have developed a deep interest and love for. This interest is the result of a cycle that has been set in motion by a yearning to learn more about it so that one can be better, which in turn infuses in one an enthusiasm to learn more just because they enjoy their achievements attained by learning.
The interesting about this is now I feel that I am in a different plane of existence. Seems like the constant spinning of the cycle has catapulted me into a different mental state, where I feel there is so much more to learn and knowledge seems never-ending. This is different from my previous plane of existence, where I was doing just what was supposed to be done, limiting myself to only knowledge of my specialization.
I must admit that at first, this new mental state left me more insecure, as its like exposing a huge waterfall to a camel thats happy drinking just the small amounts of water available in the desert. But then I realized that it is necessary to see the waterfall so that you can get an idea of how small the desert water sources are. And it is necessary to know that the water sources are small. It is necessary to know that you are ignorant....because only then can you think about achieving mastery.