Blog

Three Useful Terminal Commands
See my terminal-tools tag for more articles on nifty terminal tools.

The *nix terminal is a great tool. In everyday life this is where a lot of things can be done easy and fast with this power tool. There are an abundance of tools and ways of solving problems with the terminal, but here you have three practical applications.

Find a text string within current folder
```
grep your_magic_string -H -R *
```
The command ‘grep’ is used as a search tool for plain text files matching a regular expression. In the example above we want to find the string ‘your_magic_string’ within the current folder, so we want grep to search recursively as is done through passing with the option ‘-R’ and the wildcard (‘*’). This command will return all text files within the current folder where this text streng can be found. The -H option prints the filename.

Search and replace text within a file
```
sed -i 's/wrong/right/g' /home/ola/list-with-stuff/somethingwrong.txt
```
Sed is a stream editor that parses text and implements a programming language so that this text can be modified. The script above reads a file at the endmost location (‘/home/ola/list-with-stuff/somethingwrong.txt’) and searches through the file replacing the text pattern ‘wrong’ with ‘right’. The -i option overwrite the original file with the alterations. What used to be a negative text is now super positive.

Download a file from the Internet
```
curl -o http://http://www.lovholm.net/wp-content/uploads/2013/06/mega_secret.jpg megasecret.jpg
```
Curl is a nice tool to access resources on the Internet. If you want to see the source of a web-page just enter the URL after curl. For binaries and files you want to retain you can add the -o option. This lets you print the output to a file instead of displaying the text representation in your terminal window.

Bonus: Star Wars

All work and no play, makes ___________ (enter your name) a dull ___________ (enter your gender here). Bring popcorn, kick back in your chair and type the command: telnet towel.blinkenlights.nl. Enjoy!

Mac OSX

The mac terminal icon

What many Mac OSX users doesn’t know is that the OSX is built on Unix. In the Utilities folder under Applications you will find a terminal.Through this application you can do lots of things quick and easy. A google search reveal many good tutorials

Note: If some of the commands above prompts and error message saying that the command is unknown, you would need to install the program. Use your local packet manager such as yum or apt-get to install these. On Mac macports is a nifty packet manager.

Image: The terminal can also be used for cool stuff such as surfing the waves of the internet. The picture shows the web browser Lynx. The terminal was after all the main mode of communication with the computer before the dawn of the Graphical User Interface, and is still widely used.
30/07/2013
Push your code onto your EC2 instance with Git
Here is a little thing that has saved me a lot of work, and which I find quite neat.

I use the EC2 as my cloud computer. This page (lovholm.net) is hosted on a shared host, but sometimes it’s nice to have the flexibility to run things that are not supported by shared hosts, such as custom software and all that jazz. When I try out different web-things that are not only front-end or PHP I usually create a subdomain and place the code on an EC2-instance after initial development on localhost.

Since I now use different environments for development I have found this neat way of pushing code from my local computer to the server. This is a lot easier than using FTP, and is more flexible than deploying to Heroku (which is – by the way – a brilliant super-easy way of hosting and deploying).

The idea is very similar to that used by Heroku, except that the setup is more complex and the good terminal tool is lacking. On the other hand it gives you greater flexibility to run your own instance and it may also be more affordable if you plan to make your code into a business with much traffic.

Don’t know what git is, or don’t have git installed. Github has some great documentation on installation and Git on what git is.

Setting up the Git-repositories

Locally I initialise a git repository in the folder I am working.
```
local $ cd the-amazing-test-project
local $ git init
```
Then I use SSH to connect to my cloud computer where I have a folder for remote git repositories (I store these in a sibling-folder to the root folders of my projects.)
```
remote $ cd git-repos
remote $ mkdir testrepo.git
remote $ cd testrepo.git
remote $ git init --bare
```
When this is done, we need to create a remote-link from the local repository, and create a post-update hook at the remote repository. This will make you able to push the code from the local git-repo onto the server and from there unpack the code and do other neat things like logging and restarting of servers (if necessary).

The cloud instance as remote repository

So now you have a local repository on which you work, commit and track your local files, and you have a bare repository in the cloud. Let’s make the bridge.
```
local $ git remote add web ssh://ec2-user@yourwebserver.cloud.org/home/www/git-repos/testrepo.git
```
Create an empty file in the local directory, use
```
git add .
```
to add this file to repository, commit the changes to the git repository:
```
git commit -am "Initial test"
```
Now you could push the repository to the clouds using:
```
git push web master
```
If you get an error message. Make sure that you have added the keys used to connect to the remote host to your ssh-keys. If not, you could use ssh-add to do this. Call the command ssh-add with the key as the only argument.

If you still experience problems. Enter the local repository with your terminal and then change to the .ssh directory where you will find a file named config. This content of the file should be something similar to this:

[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = false
[remote “web”]
url = ssh://ec2-user@yourwebserver.cloud.org/home/www/git-repos/testrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*

Add a hook for deployment

Once you have a connection between the local and remote repository, we need to make a way for the remote repository to deploy our code and not just keep it safe and sound. Connect through ssh to the terminal of your remote computer and change into the git-repos directory and your testrepo.git folder.

Within this folder you should have a folder with hooks, and it is here we are going to add the last pieces of code to get our system up and running. We will utilize the server-side hook post-update to deploy our code and do other tasks needed. The post-update hook is a shell script which git executes after receiving a push. Add to this the following code:
```
#!/bin/sh
GIT_WORK_TREE=/home/www/amazing_web git checkout -f

echo "the_amazing_web_project checkin at: <$(date)> " >> /home/www/logs/updates_log
```
The second line in this code excerpt will extract the data (when adapting this snippet, make sure the path after the equal sign is created), and the fourth line will add to a logfile the name of the project and the date.

You can now commit code to your remote server easy, and if you need to do any changes like rebooting the web-server you can add the code to the post-update script.
26/06/2013
The Raspberry Pi

The Raspberry Pi has been on the market for a while now, providing a very affordable, small computer for the computer interested and hobbyist hackers. The idea of the Raspberry Pi came in 2006 when teachers at the University of Cambridge’s Computer Laboratory became concern about the decline in numbers and skill-levels among the A-level students who entered computer science. The tinkers of the early computer generation were replaced due to well designed and easy-to-use interfaces, and in school curriculum Computer Science class was mostly filled with sessions on Excel Spreadsheets, Word-processing and other tools with a high abstraction level. The idea of Raspberry Pi was to provide those who wanted with an affordable, but capable platform for tinkering. Interestingly, this trend should coincide with the makers movement which have embraced DIY-electronics from the Arduino interface to 3D-printing.

What is the Raspberry Pi?

It’s a plain computer, very scaled down and very affordable. The A-model costs around 25 Pounds, and at 30 Pounds you get the B-model which comes with 512 MB RAM, two USB-sockets and an Ethernet adapter. Both versions have a SD-card reader (the SD card, not included in the price, functions as a hard-drive for the Pi), HDMI-port for visual (and sound) and 3.5 mm jack output for sound. Both devices is powered by a micro-USB plug, come with a 700 Mhz ARM processor (roadcom BCM2835 700MHz ARM1176JZFS processor with FPU and Videocore 4 GPU) and a dedicated graphics unit which supports OpenGL and 1080p30 H.264 video (which is great since a 700Mhz processor is not too found of running high-res video in adition to the OS). It also comes with an RCA composite video output and CPIO ports, the latter may have a good potential if you want the Raspberry Pi to work as a stand-alone Arduino-like device.

Perhaps the most imporant part about the Raspberry Pi is that it can be used for many things, and that it has a low-level entry cost. With the HDMI and RCA you could use it as a media center, or you could buy a SNES-controller adapter and use it as a Super Nintendo with the ZSNES emulator. This makes it a brilliant tinkering device.

After ordering a HDMI to DVI adapter I finally got my Raspberry up and running and I will be back with some projects built on this platform soon. Stay tuned!

Getting it up and running is just plug-and-play if you have a preloaded SD-card. If not it is a bit more hazzle, I used this introduction on how to Install the Operating system onto the SD-card and it worked well using the Mac OSX Terminal to load the Raspbian OS over on the card following the instructions.

[nggallery id=11]

28/05/2013
How to set up an AWS EC2 instance
Amazon Web Service’s EC2 is the workhorse and general purpose computer in the AWS ecosystem. It is affordable (free or very cheap if you keep it scaled to the minimum option), it is easy to set up, and it provides you with a computer located in the cloud. This is a good option if you have content you would like to be accessible everywhere, and especially if you develop solutions you want to be accessible through the Internet.

Here are two examples where a cloud computer may come in handy:

1) If you have ever tried to host services on a local machine through a bread-and-butter Internet Service Provider, you will be delighted to be allocated a persistent IP-address. You can also decide on the ports you want to be accessible or shut bypassing ISP-policy, e.g. blocking outbound port 80 (default HTTP-port).

2) If you have ever used a shared host for hosting websites, you may have been met with software that doesn’t work. This can be because your scripts, plug-ins or other software are using languages not supported by the host. For example many general purpose hosts are not providing support for Python and Ruby. Another problem is libraries and other dependencies. If you want to have access to more advanced photo-editing through e.g. Imagemagick then your fate is in the hand of the provider and what stack/software/libraries/dependencies they wish to provide. You are also stuck with using FTP-client as many shared hosting providers do not grant you SSH access to their servers.

Signing up for AWS access

Setup is fairly easy. First you need to go to the AWS website and sign in or sign up with an Amazon account. As part of the registration Amazon will validate your phone number by calling you with an automatic voice service to give you a verification code, so keep your phone with you as you start the registration process. Follow through the web form for registration a Amazon account, then a

Creating the EC2 Instance

AWS has many services, but for this blog post we stick to the EC2. Click on the image to enlarge it.

Once registered and validated you should have access to the Management Console. If not you find this by first entering the AWS frontpage, then on the top of the window selecting “My account/Console” then “AWS Management Console”. This should bring up a new view with all the services that AWS provide (don’t be dazzled, there is very much information, but luckily one can survive just using a couple of them). Select the EC2 option in the orange section named “Compute & Networking”. This should bring up another view. Before continuing, go to the top right corner of the screen (to the right for your name) and select the region that is closest to you. (More on regions)

The EC2 view. From here you can create and administer your EC2 instances, and services related to these.

In the EC2-view you can easily set up and administer your instances. One thing to be aware of it that once the instance is created some settings persists with it, and these can only be changed by creating a new instance. This can cause you many hours of extra work if you do many changes to your instance, and it could also make it impossible to access the device (if you loose your encryption keys and passwords.)

There are three things we need to pay attention to at this point: what kind of instance do we want, how should we create the authentication, and how do we deal with persistant storage.

Choosing an AMI

AMI is the abbreviation for Amazon Machine Image. These are the images with the virtual appliances which are used for instanceiate the EC2 (which is basically a virtual machine). You can choose between free public AMIs, commercial images where you have to pay or you can even create an image yourself (the last is out of scope for this post). For this tutorial we are going to use a commonly used image – The Amazon Linux AMI.

From the EC2-view click the new instance button, then choose a classical view for selecting an AMI. As you can see there are several options within Linux and Windows. The ones marked with a star got a free tier if used with the micro instance, so no need to worry about the costs yet.

Creating the keys

To connect to the EC2 you will need SSH-keys which can identify you. There are two ways of creating these keys, either create the key locally and share the public key with the EC2 instance, or you can let Amazon create the keys and send you the public key for the instance. I recommend creating these locally as you then have both the private and the public keys. There seems to be a limit on downloading the keys from Amazon, and this may cause problems if you loose the key. It is also an advantage to have both keys in case you use two or more computers to communicate with your EC2 instance.

To create an SSH-key open your terminal and enter: ssh-keygen -t rsa -C “your@email.example” A tutorial is also provided by GitHub.

Upload this SSH-key when you assign keys in the EC2 setup.

Getting an IP-address

When your instance is up and running you get an access point address where you can find and connect to your instance. I would however recommend that you get an IP-address to this point. Not only is it easier and shorter, it also gives you a way of abstract your virtual machine from the address. Amazon provides you with an Elastic IP, and with this you can change which EC2-instance the IP address should point to. This makes it easy to e.g. start up a new and more powerful instance and then quickly change the IP address to this instance instead of letting the user endure the longer time a virtual reboot would take. At some point you would like to map a DNS address to the Elastic IP so e.g. sub.domain.com or domain.com points to your instance, at this point it is nice to just have to configure the elastic IP and not have to change the DNS record when you want to change instances.

Test your instance

The instance should now be up and running. Now try to connect to your instance using SSH. The login username may change from AMI to AMI, if your AMI is ubuntu the username is usually ubuntu, if you use the standard Amazon Linux AMI the username is ec2-user.
```
ssh -v ec2-user@123.123.123.12'
```
Changing the security settings

If you are not able to login it may be that you need to open the firewall to your instance. As you created your instance, you assigned the instance a security settings group.

If you have any problems connecting to a service using a specific port, you should check here to ensure that the traffic is not being blocked at the firewall. For SSH you will need to open port 22.

Creating an alias for easier access

This is not a part of the setup process, but it’s a neat little trick for making access easier. Instad of remembering the whole address in the terminal when you use SSH to connect to your instance, make an alias in the .bashrc or .bash_profile.

If you connect writing ssh -i $HOME/keys/aws.pub 1233.4556.7789.1 then put the string below into your .bash_profile and source this.
```
alias my_ec2_connect='ssh -v -i $HOME/keys/aws.pub 1233.4556.7789.1'
```
(Disclaimer to point two: lovholm.net is in fact hosted on a shared host by one.com – WordPress is very convenient to host at providers such as one, since it’s built on ubiquitous web-stack PHP+MySQL, and shared hosts are generally affordable and even easier to set-up than dedicated. One does also have a good admin dashboard from which you also can do some simple rerouting – more on this in a later post)

The image is from Flickr and is provided under a CC-licence. The image is associated to cloud computing and IBM, not Amazon.
28/04/2013
Five year anniversary

Five years ago I published the first blog post on this page, and 130 posts later I’m happy I managed my goal of writing a post now and then. Much have happened the last five years, and I hope more will happen in the five years to come. Hopefully you will find a similar post on the ten years anniversary 4th April 2017.

Thank you for reading!

Ok, I’m very enthusiastic about readers, so after a day with many readers in Google Analytics I may be like this, especially with many revisiting users. I hope to see you again:

04/04/2013
Norske illustrasjonskart på fylkes- og kommunenivå med SVG
Vektor- og rastergrafikk

SVG – Scalable Vector Graphics – er, som gitt av navnet, et skalerbart vektorbasert grafikk-markup. Vektorgrafikk er bedre egnet til skalering enn raster, da bildet ikke består av små bildeelementer (Pixel – kort for nettopp picture element), men formler for opptegning av grafikken. Nå har det seg riktignok slik at ikke alt fungerer like godt i begge grafikkverdener. Siden vektorer er basert på forskjellige punkter på et lerret og forbindelsen mellom disse, enten igjennom punkter, rette linjer, eller former for interpolerte kurver er vektorer bedre egnet til å representere abstrakte bilder og modeller, mens raster med sine pixelpunkter er godt egnet for å representere sensordata fra for eksempel bildebrikker. Det at fonter i dag representeres som vektorer gjør at disse kan skaleres uten tap i kvalitet (forsøk selv ved å forstørre denne teksten til det 10-dobbelte).

Forskjell på vektor og bitmap (raster) font.

For å overføre rasterdata til vektordata og motsatt må bildet “parses” eller konverteres, dette er ikke alltid like enkelt da bilder med mange detaljer og mye informasjon kan være vanskelig å tolke for programvaren. Har du et bilde av en rett svart strek på hvit bakgrunn er denne enkel å prosessere, men har du derimot et hus er det værre.

Et rasterbilde av et hus forsøkt vektorisert, uten hell.

SVG-formatet

En SVG formatert fil kan åpnes i flere forskjellige programmer, med ganske forskjellig visning. Dersom du åpner en SVG-fil i en leser som støtter formatet og prosesserer dette til grafikk vil du få følgende visning av kartet med norske fylker

Dersom du åpner samme fil i en teksteditor (jeg bruker textmate på mac, men du kan benytte TextWrangler som er gratis på Mac, og NotePad++ for Windows. Benytter du Linux har du helt sikkert også en favoritteditor) vil du se en struktur som i utganspunktet ser ganske uoversiktelig ut, men som ganske enkelt kan forståes. SVG-standarden har en rekke forskjellige funksjoner, men jeg kommer kun til å nevne noen i denne artikkelen.

I filene du kan benytte for å tegne kommuner og fylker på kartet er det i hovedsak paths eller stier som benyttes. Stien for å tegne Oslo kommune ser slik ut:
```
<path
id="oslo"
class="land oslo county03 ostlandet"
d="M 136.54875,466.88777 C 133.12579,463.23916 133.1611,459.6972 136.62044,459.6972 C 138.64571,459.6972 141.17041,464.85896 140.28004,467.17924 C 139.54285,469.10032 138.55185,469.02291 136.54875,466.88777 z"
style="fill:#c0c0c0;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-opacity:1"
/>
```
Verdt å merke seg her er hvordan d-taggen i elementet inneholder en ganske lang liste. Denne listen består av instruksjoner som benyttes når nettleseren, eller andre programmer tolker SVG-filen om til en visuell representasjon. M er move to. Her begynner tegneprosessen. C er kurve. Hit sendes tre punkter som videre benyttes for å interpolere en kurve. Dette fortsetter fram til formen sluttes med kommandoen Z. For å markere et fylke benytter vi fill-argumentet i style-taggen. Som du kan se består denne av en grå fargeverdi nå (elementert kjære Watson, både R – to første etter hashtag, G – to i midten – og B – to til høyre – har samme hexadecimale verdi. RGB)

Norske fylker og kommuner

Ok, nå har vi noe kunnskap om SVG, men hvordan er det vi får markert enten fylker eller kommuner med fine farger? Jo, det neste vi trenger er SVG filer med Norgeskart.

Wikipedia har en fin SVG-fil over norske kommuner. Høyreklikk på bildet og last ned fila.

Wikipedia har også en fin SVG-fil over norske fylker. Høyreklikk på bildet og last ned fila.

Finn noen fine farger. Dersom du ikke har noen favoritt RGB-hex fargekoder anbefaler jeg Adobes gode Kuler verktøy.

Editer SVG-fila

Filene du nå har lastet ned inneholder path elementer for enten fylker eller kommuner. Ta fila for det politiske nivået du ønsker, åpne denne filen i din favoritteditor, søk deretter på navnet på kommunen eller fylket og skriv inn en annen farge for kommunen eller fylket du ønsker å endre. For eksempel kan du ta utgangspunkt i tekstutsnittet av Oslo kommune fra ovenfor (Oslo er både fylke og kommune, og vil derfor være i begge filene. Merk: Dette er Oslo fra kommunefilen) og legg inn en annen fargekode for fyll.
```
<path
id="oslo"
class="land oslo county03 ostlandet"
d="M 136.54875,466.88777 C 133.12579,463.23916 133.1611,459.6972 136.62044,459.6972 C 138.64571,459.6972 141.17041,464.85896 140.28004,467.17924 C 139.54285,469.10032 138.55185,469.02291 136.54875,466.88777 z"
style="fill:#FF0000;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-opacity:1"
/>
```
Du vil nå ha et rødt Oslo.

Et svart Hedmark, grønt Akershus og rødt Finnmark.

Et grønt Rendalen (Norges største kommune utenfor Finnmark) og Hamar.

Bilder:

Bilde for å illustrere forskjellen på vektor og raster font, samt bilde av dårlig vektorisert hus er lånt fra.

http://www.1stwebdesigner.com/design/pixel-vector-graphics-difference/

http://giorgioscappaticcio.wordpress.com/category/task-1-image-sourcing/
06/03/2013
Enter the Cloud – Amazon Web Services

AWS – Amazon Web Services – is a great infrastructure cloud service for all IT business from start-ups to full blown corporations. Providing both Infrastructure-as-a-Service (IaaS) through the general purpose EC2 and Platform-as-a-service (PaaS) through Beanstalk as well as hybrid solutions such as storage solution S3 and database service RDS.

I have been using these services for about half a year now, and I am convinced these services as well as competing equals (e.g. Microsoft Azure) is a good tool that for everyone who want to experiment with web technology or create web businesses. This is not a new thing, cloud computing have been around for years, and many web services are running on Amazon services today, but through looking into them myself I understood how I can benefit from them in my work with the web and through the Internet, and new experiments

S3

Amazon Simple Storage Service provide affordable web space. In this specific market similar services have emerged, which also are easier accessible e.g. Dropbox – one of the best things since bread came sliced. The good thing about S3 is the tight integration with the other Amazon services, it is also good for hosting static web pages (pages that only needs to be served, without initial computation). Some FTP clients also include S3 support so you can use the S3 service as you would regularly use FTP for storage of files. Another advantage is that you can choose the S3 bucket (the entity in which you place your files on S3) to be distributed to all of Amazons data centres (for an additional cost). This is good if you need to transfer files to end user fast as there are less potential for bottlenecks using a local datacenter (by local here I mean the same continent.)

EC2

Amazon Elastic Compute Cloud is the real game changer. While most of the other services are specialised the EC2 is the general purpose machine. An EC2 instance is a virtual computer located in the cloud where you pay for the computing hours you use, a standardised unit being approximately: “CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.” Since the payment is by the hour, and the units are scalable it can be very flexible to your needs. The service enables you to make your hosting scalable both for scaling up, and scaling out.

Scaling up: The instances comes in several packages. From the modest and almost free micro instance to extra-large distributed platforms. You can change the processor performance and dedicated RAM on your instance, and changing the instance type just require a reboot and you can transfer your persistance storage onto the new virtual machine.

Scaling out: If you create an image with all your server variables, binaries, libraries and other stuff you need, or a method for loading this from a remote computer, you can in just a couple of minutes initialise new instances with these data, and since you pay by the hour you can e.g. have four computers running at times with high load and one during times with low load. The coolest feature is that you can measure trafic and load new servers as the service is running at lower performance than wanted.

Elastic-IP:

The URL to the instances are normally quite arcane and difficult to remember. That is why this is quite good. Elastic-IP is just bridge between a IPv4 address that you get allocated, and the instance. Above I described how you could reboot an instance, but the downside to this is that while the instance is restarting your server would be down. To avoid this, you could create a clone of your instance #1 with the higher capacity settings, and when this is loaded, direct the IP to the new instance #2 and then turn off the instance #1. (Well, I guess you would loose the users’ sessions)

Do you want to set up an EC2 instance? (More about how)

RDS

RDS, or the Relational Database Service. You need this if you want to handle persistent data in a relational manner (and if you’re not hacking user-data into self-composed formatted files, or using a noSQL solution) and think life is to short to set-up and configure a SQL-database on the general purpose EC2 service.

When you create a new DB instance you can easily choose type. If you use Oracle you can bring your own licence or rent licence by a hour-licence.

For the RDS you can choose between (ex-Swedish, now Oracle) MySQL, Microsoft SQL Server or Oracle (also Oracle *dough* ). The setup is very easy, and you can choose the size of the database, the backup retention period and other variables then start the instance. Check the security settings that the ports are open and that the DB can access traffic from your IP-address. Download SQL Workbench (MySQL) or SQL Developer (Oracle) (Sorry, not sure about good tool for MS SQL Server), insert the endpoint URL, username and password and get started.

Other services

The AWS selection is huge, but luckily you won’t need to know every service available to get started. (Click images for full size)

This picture show the AWS panel from which you administer your services, and as you can see there are several more services to explore. Many of these are specialised to e.g handle MapReduce or deploying specific web-stacks easily, but with the S3, RDS and EC2 it should be easy to get started.

22/02/2013
Work programmatically with Google Spreadsheets

Some time back I authored a script which reads through a CSV formatted list, and based on the artists’ names tried to decide the nationality of the artists by querying the last.FM search engine and parsing the XML structured result.

The script worked, and found and returned about 80% of the artists, and around 80% of these again had the conceptually similar artist. What if the alteration could be done in the document itself? For the CSV based script the artists had to be extracted from the database, then parsed and then put into the document. With this attempt we can skip the middle step, and let the script run simultaneous with data being available to view for the users.

If you haven’t already used Google Docs, you should consider it, as it is a very convenient way of working with documents, especially where there are more users. Through Google Docs you can collaborate on working on documents, and since they are stored in clouds all users will instantly have the latest version. Another advantage is that whey you are working on the document simultaneously each user is editing the same document, so merging different versions is not a problem.

A great tool for combining Google Spreadsheets with the Python programming environment is the gspread module. After installing this package you only need to import the code and then you can with very few lines of code retrieve and update information in the spreadsheet cells.

The specifics for doing this task is pretty much these few lines. Import the package. Login, find correct document and do whatever you need to. (this excerpt won’t work. Check the whole script under – NB: mind the indentation. It may not be correct displayed in the browser.)

[sourcecode language=”python”]
import gspread

G_USERNAME = ‘your@gmail.com’
G_PASSWORD = ‘yourPassword’
G_IDENTIFIER = ‘document_identifier_checkdocument_url_in_your_browser’

# Connecting to the data source
gc = gspread.login(G_USERNAME,G_PASSWORD)
sht1 = gc.open_by_key(G_IDENTIFIER)
worksheet = sht1.get_worksheet(0)

for a in range(2,5124):
try:
name = worksheet.acell(get_val[a]).value
res = getCountry(name)
if res:
print res
country, last_id, name = res
worksheet.update_acell(set_name[a], name)
worksheet.update_acell(set_country[a], country)
except Exception as e:
print e

[/sourcecode]

Above is the lines related to connecting Python to the google docs, under you can see the whole script and how the method I mentioned in a post earlier is used in this setting.

[sourcecode language=”python”]
#!/usr/bin/python
# -*- coding:utf-8 -*-

"""
Clouds & Concerts – 2012
Ola Loevholm

Initialized from the commandline. Runs through The Google doc spreadsheet with topp 5000 artists, and
runs the parsing query against the Last.FM browser then enters the country and search string (for validation)
into the google docs.

"""

G_USERNAME = ‘your@gmail.com’
G_PASSWORD = ‘yourPassword’
G_IDENTIFIER = ‘document_identifier_checkdocument_url_in_your_browser’

import sys, urllib, string, csv, time
import xml.etree.ElementTree as ET
import gspread

# Loads a dictionary with ISO 3166-1 abbreviations and codes
COUNTRIES = {"AF":"AFGHANISTAN","AX":"ÅLAND ISLANDS","AL":"ALBANIA","DZ":"ALGERIA","AS":"AMERICAN SAMOA","AD":"ANDORRA","AO":"ANGOLA","AI":"ANGUILLA","AQ":"ANTARCTICA","AG":"ANTIGUA AND BARBUDA","AR":"ARGENTINA","AM":"ARMENIA","AW":"ARUBA","AU":"AUSTRALIA","AT":"AUSTRIA","AZ":"AZERBAIJAN","BS":"BAHAMAS","BH":"BAHRAIN","BD":"BANGLADESH","BB":"BARBADOS","BY":"BELARUS","BE":"BELGIUM","BZ":"BELIZE","BJ":"BENIN","BM":"BERMUDA","BT":"BHUTAN","BO":"BOLIVIA, PLURINATIONAL STATE OF","BQ":"BONAIRE, SINT EUSTATIUS AND SABA","BA":"BOSNIA AND HERZEGOVINA","BW":"BOTSWANA","BV":"BOUVET ISLAND","BR":"BRAZIL","IO":"BRITISH INDIAN OCEAN TERRITORY","BN":"BRUNEI DARUSSALAM","BG":"BULGARIA","BF":"BURKINA FASO","BI":"BURUNDI","KH":"CAMBODIA","CM":"CAMEROON","CA":"CANADA","CV":"CAPE VERDE","KY":"CAYMAN ISLANDS","CF":"CENTRAL AFRICAN REPUBLIC","TD":"CHAD","CL":"CHILE","CN":"CHINA","CX":"CHRISTMAS ISLAND",
"CC":"COCOS (KEELING) ISLANDS","CO":"COLOMBIA","KM":"COMOROS","CG":"CONGO","CD":"CONGO, THE DEMOCRATIC REPUBLIC OF THE","CK":"COOK ISLANDS","CR":"COSTA RICA","CI":"CÔTE D’IVOIRE","HR":"CROATIA","CU":"CUBA","CW":"CURAÇAO","CY":"CYPRUS","CZ":"CZECH REPUBLIC","DK":"DENMARK","DJ":"DJIBOUTI","DM":"DOMINICA","DO":"DOMINICAN REPUBLIC","EC":"ECUADOR","EG":"EGYPT","SV":"EL SALVADOR","GQ":"EQUATORIAL GUINEA","ER":"ERITREA","EE":"ESTONIA","ET":"ETHIOPIA","FK":"FALKLAND ISLANDS (MALVINAS)","FO":"FAROE ISLANDS","FJ":"FIJI","FI":"FINLAND","FR":"FRANCE","GF":"FRENCH GUIANA","PF":"FRENCH POLYNESIA","TF":"FRENCH SOUTHERN TERRITORIES","GA":"GABON","GM":"GAMBIA","GE":"GEORGIA","DE":"GERMANY","GH":"GHANA","GI":"GIBRALTAR","GR":"GREECE","GL":"GREENLAND","GD":"GRENADA","GP":"GUADELOUPE","GU":"GUAM","GT":"GUATEMALA","GG":"GUERNSEY","GN":"GUINEA","GW":"GUINEA-BISSAU","GY":"GUYANA","HT":"HAITI","HM":"HEARD ISLAND AND MCDONALD ISLANDS",
"VA":"HOLY SEE (VATICAN CITY STATE)","HN":"HONDURAS","HK":"HONG KONG","HU":"HUNGARY","IS":"ICELAND","IN":"INDIA","ID":"INDONESIA","IR":"IRAN, ISLAMIC REPUBLIC OF","IQ":"IRAQ","IE":"IRELAND","IM":"ISLE OF MAN","IL":"ISRAEL","IT":"ITALY","JM":"JAMAICA","JP":"JAPAN","JE":"JERSEY","JO":"JORDAN","KZ":"KAZAKHSTAN","KE":"KENYA","KI":"KIRIBATI","KP":"KOREA, DEMOCRATIC PEOPLE’S REPUBLIC OF","KR":"KOREA, REPUBLIC OF","KW":"KUWAIT","KG":"KYRGYZSTAN","LA":"LAO PEOPLE’S DEMOCRATIC REPUBLIC","LV":"LATVIA","LB":"LEBANON","LS":"LESOTHO","LR":"LIBERIA","LY":"LIBYA","LI":"LIECHTENSTEIN","LT":"LITHUANIA","LU":"LUXEMBOURG","MO":"MACAO","MK":"MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF","MG":"MADAGASCAR","MW":"MALAWI","MY":"MALAYSIA","MV":"MALDIVES","ML":"MALI","MT":"MALTA","MH":"MARSHALL ISLANDS","MQ":"MARTINIQUE","MR":"MAURITANIA","MU":"MAURITIUS","YT":"MAYOTTE","MX":"MEXICO","FM":"MICRONESIA, FEDERATED STATES OF",
"MD":"MOLDOVA, REPUBLIC OF","MC":"MONACO","MN":"MONGOLIA","ME":"MONTENEGRO","MS":"MONTSERRAT","MA":"MOROCCO","MZ":"MOZAMBIQUE","MM":"MYANMAR","NA":"NAMIBIA","NR":"NAURU","NP":"NEPAL","NL":"NETHERLANDS","NC":"NEW CALEDONIA","NZ":"NEW ZEALAND","NI":"NICARAGUA","NE":"NIGER","NG":"NIGERIA","NU":"NIUE","NF":"NORFOLK ISLAND","MP":"NORTHERN MARIANA ISLANDS","NO":"NORWAY","OM":"OMAN","PK":"PAKISTAN","PW":"PALAU","PS":"PALESTINIAN TERRITORY, OCCUPIED","PA":"PANAMA","PG":"PAPUA NEW GUINEA","PY":"PARAGUAY","PE":"PERU","PH":"PHILIPPINES","PN":"PITCAIRN","PL":"POLAND","PT":"PORTUGAL","PR":"PUERTO RICO","QA":"QATAR","RE":"RÉUNION","RO":"ROMANIA","RU":"RUSSIAN FEDERATION","RW":"RWANDA","BL":"SAINT BARTHÉLEMY","SH":"SAINT HELENA, ASCENSION AND TRISTAN DA CUNHA","KN":"SAINT KITTS AND NEVIS","LC":"SAINT LUCIA","MF":"SAINT MARTIN (FRENCH PART)","PM":"SAINT PIERRE AND MIQUELON","VC":"SAINT VINCENT AND THE GRENADINES",
"WS":"SAMOA","SM":"SAN MARINO","ST":"SAO TOME AND PRINCIPE","SA":"SAUDI ARABIA","SN":"SENEGAL","RS":"SERBIA","SC":"SEYCHELLES","SL":"SIERRA LEONE","SG":"SINGAPORE","SX":"SINT MAARTEN (DUTCH PART)","SK":"SLOVAKIA","SI":"SLOVENIA","SB":"SOLOMON ISLANDS","SO":"SOMALIA","ZA":"SOUTH AFRICA","GS":"SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS","SS":"SOUTH SUDAN","ES":"SPAIN","LK":"SRI LANKA","SD":"SUDAN","SR":"SURINAME","SJ":"SVALBARD AND JAN MAYEN","SZ":"SWAZILAND","SE":"SWEDEN","CH":"SWITZERLAND","SY":"SYRIAN ARAB REPUBLIC","TW":"TAIWAN, PROVINCE OF CHINA","TJ":"TAJIKISTAN","TZ":"TANZANIA, UNITED REPUBLIC OF","TH":"THAILAND","TL":"TIMOR-LESTE","TG":"TOGO","TK":"TOKELAU","TO":"TONGA","TT":"TRINIDAD AND TOBAGO","TN":"TUNISIA","TR":"TURKEY","TM":"TURKMENISTAN","TC":"TURKS AND CAICOS ISLANDS","TV":"TUVALU","UG":"UGANDA","UA":"UKRAINE","AE":"UNITED ARAB EMIRATES","GB":"UNITED KINGDOM","US":"UNITED STATES",
"UM":"UNITED STATES MINOR OUTLYING ISLANDS","UY":"URUGUAY","UZ":"UZBEKISTAN","VU":"VANUATU","VE":"VENEZUELA, BOLIVARIAN REPUBLIC OF","VN":"VIET NAM","VG":"VIRGIN ISLANDS, BRITISH","VI":"VIRGIN ISLANDS, U.S.","WF":"WALLIS AND FUTUNA","EH":"WESTERN SAHARA","YE":"YEMEN","ZM":"ZAMBIA","ZW":"ZIMBABWE"}

 

# Connecting to the data source
gc = gc = gspread.login(G_USERNAME,G_PASSWORD)
sht1 = gc.open_by_key(G_IDENTIFIER)
worksheet = sht1.get_worksheet(0)
# Iterates through XML-structure and removes the namespace, for easier navigation in getCountry()s ElementTree.findall()
def remove_namespace(doc, namespace):
"""Remove namespace in the passed document in place."""
ns = u'{%s}’ % namespace
nsl = len(ns)
for elem in doc.getiterator():
if elem.tag.startswith(ns):
elem.tag = elem.tag[nsl:]
# getCountry – where the magic happens. Encodes string with artistname to url, then query musicbrainz search engine.
# parses the XML-answer and get the name, id and country of the first returned element (with highest weight)
# returns country name i a) artist is found through the search engine b) artist has a country associated to the profile, otherwise returns False
def getCountry(name):
name = urllib.quote_plus(name)
BASE_URL = "http://musicbrainz.org/ws/2/artist/?query=%s&format=xml&method=advanced" % (name)
print "Querying: %s" % (BASE_URL)
try:
search_input = urllib.urlopen(BASE_URL)
# Checks whether HTTP Request Code is 200 – if not goes to sleep for 5 seconds // Inded for 503 Code
http_code = search_input.code
if http_code != 200:
# print "Could not access: %s \t Got HTTP Code: %s. 5 second cool-down" % (name, http_code)
time.sleep(5)
getCountry(name)
except Exception:
print "GETTING_ERROR: Something went wrong while getting HTTP"
return False
#search_xml = search_input.read()
#print search_xml
try:
tree = ET.parse(search_input)
remove_namespace(tree, u’http://musicbrainz.org/ns/mmd-2.0#’)
feed = tree.getroot()
elem = feed.findall("./artist-list/")
#print elem[0].find(‘name’).text
#print elem[0].get(‘id’)
except Exception:
print "PARSE_ERROR: Something went wrong while parsing HTTP"
return False
try:
if elem[0].find(‘country’) != None:
# print COUNTRIES[elem[0].find(‘country’).text]
try:
country = COUNTRIES[elem[0].find(‘country’).text]
except Exception:
print "Could not find key in countrylist error"
return False
return [country,elem[0].get(‘id’),elem[0].find(‘name’).text]
else:
print elem[0].find(‘name’).text + " has not any country associated\n"
return False
except IndexError, ValueError:
print "ERROR – COULD NOT GET DATA FROM %s\n" % (name)
return False

#Running through the values
get_val = []
set_name = []
set_country = []
for a in range(2,5124):
v = "B%s" % a
sn = "H%s" % a
sc = "G%s" % a
get_val.append(v)
set_name.append(sn)
set_country.append(sc)

for a in range(2,5124):
try:
name = worksheet.acell(get_val[a]).value
res = getCountry(name)
if res:
print res
country, last_id, name = res
worksheet.update_acell(set_name[a], name)
worksheet.update_acell(set_country[a], country)
except Exception as e:
print e

[/sourcecode]

03/02/2013
Facebooksider som RSS
RSS er super-praktiskt. Enten om du bruker laptop, tablet eller mobil så er aggregerte kilder gull verdt. Hvor digg er det ikke å kunne samle alt du ønsker å lese i en leser (f.eks Google Reader) eller til og med automatisk oppdatere nettbrettet ditt med de nyeste sakene fra dine favorittkilder?

Her er en liten oppskrift på hvordan du kan gjøre dette på facebook sider du følger. Det er ikke alltid at social feed plukker opp disse, og dessuten er det fint å kunne lagre oppdateringer for ettertiden.

Det første du trenger er denne URL strengen:
```
https://www.facebook.com/feeds/page.php?format=atom10&id=241453889260137
```
Denne ganske ordinære URL strengen som parses som to argumenter av webserveren [:format => “atom”, :id => “241453889260137”] (ruby hash syntax), et voila, du får en atom formatert RSS side tilbake. Det du gjør her er å bytte ut id fra URLen ovenfor med iden fra siden du ønsker å følge.

Men hva om det ikke er noe ID i side URLen på Facebook? Som f.eks i bildet under (tatt fra siden til Løvholm Digitale Medier – mitt forholdsvis nye enkeltmannsforetak).

Her blir det litt mer komplisert.Dersom du forsøker å laste inn atom RSS-feeden fra dette navnet vil du kun få en side med en feilmelding.

Vår vei til målet går igjennom fantastiske Open Graph, teknologien Facebook bruker for å organisere sine tjenester og sin verden, og dens API.
```
http://graph.facebook.com/lovdigmed
```
Her kan du bruke navnet som oppslag og få tilbake en JSON-formatert tekststreng med det du trenger å vite

Nå trenger du kun å gå tilbake til den første URLen vi hadde, og benytte IDen til siden (merk: ikke cover_id) for å få tilgang til RSS-feeden.

Nå kan du lese Facebook sideoppdateringer i din yndlingsleser sammen med alt annet interessant du abonnerer på
30/01/2013
New layout for lovholm.net

As the old layout broke with the newer versions of WordPress, and the fact that it was on high time to get a new layout (the last major visual update was back in 2010 – very long ago in the web world) I have now changed for a beta version of the Arras theme.

As you may see from the current state, not everything is up to date, but I hope you have patience as I will now start to customise the theme (maybe even a post on WP PHP programming will follow).

For future updates, stay tuned 🙂 – or subscribe to the RSS feed.

30/01/2013