University posts

You may notice that every now and then there appears a post that seems forced, or not in line with the kinds of posts I usually make such as the Javascript and DOM references post. This is because one of my university courses at Bond University is running a Blog Assessment in which we are given certain topics to blog about every now and then. Since these assignments are compulsory, I cannot refrain from writing them though they clutter the rest of the blog entries.

Oh well, just thought I would explain the issue for those who are confused by these seemingly random posts.

My web

We all use the web differently, and all have the pages we cannot live without. Following is a list of the top websites I cannot live without. What are yours?

Google Apps

Google Apps is a service that allows you to manage Google services for your own domains. For instance – I own several domains (thesquareplanet.com, casa-rioja.com, awknard.com, nerd-geek.com …), and host them all myself. I also used to host my own mail and calendar, but when I discovered Google Apps, I stopped doing that. Not only would I lose all mail that was sent to me if my server or internet connection went down, but I also had to constantly maintain security and other enhancements. With Google Apps, you can add all of your domains into a central system, and manage e-mail addresses, users, calendars and all sorts of other Google hosted services. Best thing is – it’s free! (as long as you’re non-profit that is)

I now have the GMail and GCalendar interfaces on my own domains, and can use all of their features as if I was using my Google Account.

Facebook

Although Facebook receives quite a bit of critique with regards to privacy issues, and many abandon it entirely because of this, I feel that it is truly an amazing tool on the web that should be taken advantage of. As with everything else one does online, one has to be conscious that everything that is put out can, and probably will, be used against you, but as long as this is clear, Facebook brings a lot to the table. It allows you to organize your social life, and easily stay in touch with everyone you know (and some you don’t). By using Facebook, you also create an online presence which is extremely important in this increasingly online world.

Google Reader

If you follow anything online, be it a blog, a forum, a news site, a video feed or acoolname.com, you should get a feed reader. It allows you to merge feeds from all your article sources, and view them in one grand, unified list; sorted and grouped to heart’s desires. Google Reader is an online feed reader which provides a clean online interface for you to check updates in your online world from anywhere, at any time. All you need is a Google account, and you’re on your way!

What else do I use?

• Vimeo – Upload & watch videos
• YouTube – Upload & watch videos
• GrooveShark – Listen to music online for free
• Songza – Listen to music online for free
• XMarks – Synchronize your bookmarks and passwords across browsers and computers, and access them from anywhere
• StumbleUpon – Find new, interesting pages on the web

SPeeDY – The new HTTP?

Google have been coming with a lot of new cool projects lately – From Chrome and Chrome OS to Wave and Social Search. Following this innovative trend, they have now announced that they’re working on a possible replacement for HTTP. Actually, it is not as much a replacement as it is an augmentation or “fix”. SPDY will still be using the headers and basic structure of HTTP, but will treat that structure quite differently and introduce several enhancements to make it more efficient and more suitable for the contemporary web context.

Google argues that we need a new protocol for web traffic because of the way the web has changed over the last decade. Nowadays, pages use both multimedia and several externally linked files – something which HTTP was not optimized for. More specifically – HTTP does not allow:

• Fetching several resources through a single HTTP connection
• Push-like behavior from the server to the client
• Lack of native, and compulsory, compression of packet contents – especially headers
• Statelessness – HTTP does not “remember” anything from previous exchanges. This makes for redundant information such as certain almost static headers to be resent unnecessarily

Although several other projects have tried to come up with an appropriate replacement for the HTTP protocol, Google now believe that they have found one that is suitable. The SPDY project aims to accomplish several things; for example, a 50% decrease in latency for online request-response cycles and near-transparent transfer from the old technology to the new one.

This latter point is quite interesting, and has been the reason why many other similar proposals have failed. Far too many protocols attempt to reinvent the wheel, but Google has decided to retain the TCP protocol as the underlying transportation agent, and to minimize the impact on developers and end-users. This is achieved by the SPDY protocol avoiding any changes to the way the data is handled on both end-points, only the protocol in between, so that web developers won’t have to change a thing on the server side. The only changes that are needed are in the browsers and the web server itself.

The reduction in latency will be primarily by enforcing compression on headers and body, slicing away unnecessary header tags and allowing several resources to be fetched in a single TCP request to avoid packet overhead. Google has also decided to take some steps to improve the overall quality of the protocol as a whole by introducing SSL as the rule, and non-SSL as the exception (if it will be allowed at all); as well as cutting down the protocol definition so that the implementation will be much simpler.

Overall, SPDY promises a lot, and looks very promising – All that remains is to see whether server and browser developers will join the cause and develop working implementations of the draft for testing. Knowing Google, they will probably release support for it in both Chrome and an “experimental” web server that will probably be released soon.

One major obstacle for its popularity though is that the premise of multiple resources in a single TCP stream, and a move away from the stringent request/response cycle of current HTTP sessions means substantial changes will be made to the web servers to allow for this kind of behavior. Hopefully Google will release their testing server to the public soon so we can start to see test implementations of the technology, and how hard or easy it will be to implement.

For more info on the technology, have a look at the SPDY Whitepaper

Surveying Social Search

Google recently published a new feature on their Labs Search Experiments page: Google Social Search. By enabling this service, Google attempts to locate your friends and connections through online communities like Twitter, Facebook, GMail and Google Reader, and present you with search results that are not only relevant to your search phrase, but also to your social circle. For instance, searching for a restaurant will provide you not only with the regular search results for that restaurant, it will also give you blog posts, tweets and other publicly available messages from your friends, and their friends about what you search for. This may be a review, a comment, images or anything else that Google finds relevant.

In order for this to work correctly, you must tell Google how to find your friends, and this is usually done through your Google Profile. There, you can add links to pages that represent you online, and Google will then go through these to attempt to find connections to other people. Results from these people will then be integrated into your search results.

It is hard to say what kind of an impact this will have on searching in the future, but I believe it will make search results much more relevant to the individual. We are more likely to listen to someone we know about a product or a service, than believe the words of a total stranger. The real roadblock that needs to be overcome is to properly determine a user’s contacts and friends. Because several systems are closed to the public (Facebook for instance), a lot of connections are invisible to these kinds of crawlers. In my opinion, this is very unfortunate, and I think it is sad that people feel such a need for privacy that they feel the need to hide their friends, pictures and info from the public eye. As Google Social Search is currently dependent solely on publicly available connections and information, it cannot get any better than what is publicized. It is limited by the raw data it can fetch, and at the moment, that is unfortunately too little for the social search results to be relevant enough. That is, at least, my experience so far with trying out Google Social Search. Hopefully, however, this will change in the future, and bring truly individually adapted search results.

Jump-starting tricks for aspiring web developers

So, you want to make websites, do you? Becoming a web developer is both very easy, and very hard at the same time. Mocking up a simple page online with some text and images is easy. Not only are there several WYSIWYG website editors (What You See Is What You Get) out there, but there are also several websites that allow you to create your page online directly through point and click. This is not web development.

Furthermore, if all you want to do is make a design for a web page in Photoshop, you are not a web developer, you are a web designer. Although many web developers tend to be web designers and vice-versa, this is certainly not a matter of implication. A web designer creates a site design, a web developer implements that design – there is nothing more to it. If you create your designs and implement them, you are a web developer as well as a web designer.

There are, of course, several degrees of web development; from basic HTML and CSS to fully-fledged PHP/ASP/<insert programming language here> web applications, but this is not the topic of this post. In this post I intend to give you, as an aspiring web developer, a couple of shortcuts, strategies, tricks and gotchas that I have found during my six years of development experience at the time of writing. This is by no means a complete guide to becoming a web developer, but more of a reference document to get you past the various obstacles browser developers, web standards, faulty documentation and operating systems have put in place to make our life a bit more interesting.

So, without further ado:

The only 5 tags you’ll ever need

HTML and XHTML both contain large amounts of tags. Too many, in fact, for them all to be useful in most cases. Remember, XHTML (and HTML to a large degree) aims to describe the structure of the content, and that is what we have all the tags for. To make the content readily available to screen-readers, text-to-speech engines, search engines and the likes. When prototyping a design, however, you should rarely make use of all of, for instance, the subtle difference between an <strong> and a <b> tag. In fact, you probably shouldn’t even care about the difference between an <em> and a <strong>; you would probably substitute them both with a <span> anyway. Although you should put in the appropriate tags when you begin to develop larger websites, or when you begin to polish the smaller ones, you will find that there are only 5 tags you really need when building an initial design.

• <span> – The fundamental inline element
• <div> – The fundamental block element
• <a> – The link
• <img> – The image
• <ul> – For making lists

Although coding your site using only these tags may be considered bad practice for the reasons explained above, they will in fact get you started quickly and substantially reduce the amount of tags you have to keep in your head when you’ve just started making web pages.

Due note that these are only content tags, and not the additional meta tags such as <link>, <style>, <script> and <meta> that you will also need to style and animate your site.

Reset your styles

The #1 reason why your designs do not work when you try to open them in a different browser from the one you initially developed and tested in is because of default margins and paddings. Every browser has its own definition for what padding and margin every element should have if you don’t specify any, and consequently, when you move to another browser, all your elements become slightly smaller or larger, and your design collapses into an unrecognizable heap of divs for no apparent reason.

Although reset stylesheets (google it) have recently become quite popular, I often find them unnecessary as they tend to reset too much – giving you more work. Instead, I just a good old rule which simply resets the margin and padding, and nothing more:

* { margin: 0; padding: 0 }

Try putting this in your document before you begin, and you’ll find cross-browser design becomes a whole lot easier!

Understand the box model

Way too many web developers don’t understand what the difference between margin and padding is, and how these are rendered together with the border of the element. Much less how to calculate the total dimensions of the element. The fact of the matter is that this is essential to being able to create potent web sites. It is also the alpha-omega of many of the CSS hacks you will encounter through your web development career.

A simple Google search reveals several images and sites trying to explain it, and one of the first results explains it quite simply:

Learn it by heart – it will save you much hassle and confusion!

Face the truth – learn to program

If you’re going to make any decent web site, you will have to learn how to program. And I’m not talking about plain ol’ HTML, I’m talking of at least Javascript, and preferably a proper server-side language such as PHP or ASP. Javascript allows you to manipulate your site dynamically to make your site a lot more interactive. For instance, you can use it to show a date-picker for an input field, validate form input without going through the server (though for security reasons you should ALWAYS check the data on the server as well), update the page behind the scenes without the user having to refresh the page (AJAX) or make elements on your page fly all over the place. All this power, however, becomes nothing but a fun topping when you consider a server-side language.

Where Javascript deals with the page the user is seeing, the server-side languages allow you to store data the user submits, add dynamic content to your site (This can be anything from a simple “Quote of the day” to allowing users to add articles, list recently added articles and show all articles using the same HTML, allowing the scripting language to fill in the content) and track user sessions (i.e. login and have their own preferences and personalized pages).

Personally, I prefer PHP to ASP, but this is completely up to you – Just learn one because you will have to!

Use a JavaScript library

Javascript is a wonderful thing, but it also quite awkward. In order to do even simple animations, you need to write a lot of code. In addition, Javascript lacks the broad set of tools that often comes with larger, self-contained languages. There are many libraries out there that attempt to “fix” Javascript in one way or another. Some simply make it easier to manipulate the DOM (Document Object Model) and do animations (jQuery is a typical, and extremely popular library that aims to do this), whilst others go all the way and manipulate Javascript’s native methods and objects to make them more powerful, more usable and more flexible. A good example of the latter is the Javascript framework/library MooTools. For larger, more complex Javascript-enabled projects, such a framework is often preferred over the lightweight jQuery equivalents. Pick the tool for the task at hand.

Know your clients

When developing a website, you need to know what browsers and resolutions you are developing for. If you are making a site for a design bureau, you can usually assume that they will have high resolution screens of at least 1280×1024 or 1600×1400, and you should design your site accordingly. In such cases a fluid design layout might be worth considering to allow your users to utilize the full resolution of their screens. More commonly though, you will be designing for the majority of users, and the majority do not have resolutions of that scale. Too many users still use a 1024×768 resolution, or even smaller, and consequently we have to take this into consideration. Usually, making a page between 900px and 950px wide makes the site viewable for most users, and at the same forces you to avoid extraneous information.

Also, determine whether you should support IE6 or not. This is a major issue as supporting IE6 requires a lot more work than not supporting it because of its blatant disregard for standards and ridiculous implementations of it at times.

A word of caution: If a designer hands you a design with a page-width wider than 950px, don’t just scale it down and get on your way! This will not only distort the design, but it will also make all the text smaller which you should avoid at all costs. Make sure all text on your site is easily readable, and try to keep to a maximum of 15 words per line. Anything smaller makes it hard to read, especially on high resolutions!

A second word of caution: Not everyone has Javascript enabled! Make sure you either provided a usable scaled-down version of your site without JS, or that you warn the user that your site requires Javascript. Don’t just leave it up to them to find that nothing works..

Prototype, then validate

If you’re told to make a design, make a quick and dirty mock-up. That is not to say it should not look like the design, but that you shouldn’t care too much whether your prototype validates or uses the correct tags. Those kinds of things can be changed later if the design is approved. Designers change their mind all the time, and you don’t want to spend a lot of time on something they are going to change or remove entirely at the next signpost.

When you’re prototyping, however, do try to make the site look similar in all target browsers. The reason for this is mainly because if you make it work in one browser and OK the design, and then, when told to make the site to the design you OK’d, you might find that what you did in that one browser cannot be done in another due to different implementations of the standard. Then what are you going to do? You OK’d the design, remember?

Follow standards, but not blindly

Standards are a good thing – no question about it. The problem is that at times, it can be a bit too strict. Especially when trying to make your pages render correctly cross-browser.

For instance, IE6 only supports the CSS :hover attribute on <a> elements, and as such, you may be force to put divs, or other block-level elements, inside an <a> tag which will not validate. It will seldom create any problem in any other browser, so theoretically you could just ignore the validation warnings. The problem is that all too many people are concerned about whether their code validates or not. Bottom line is, if it works, and you’re confident that it works, and will continue to work cross-browser, the warning is really not that important.

That said, if you can follow the standard and validate your site, do so. It is a form of guarantee that your site design should not be broken by future browsers that might have different rendering engines or follow newer standards. Often, when you’re code doesn’t validate, it is quite simple to fix the issue. The fix also tends to come at the expense of a degraded experience to IE6 users…. Oh well, that’s sad isn’t it?

Plan for security, but delay the implementation

When developing web applications, security should be a major concern. Unfortunately, it is often completely overlooked, or applied seemingly haphazardly; “Oh, I think I’ll just put in a striptags here and we’ll be good to go!” Not thinking about, and planning, for security can cost you very dearly in the end.

On the other hand, security takes a lot of time and effort to implement, and in the initial stages of web development – when prototyping a new concept or design – you don’t really want to bother with that kind of thing. The danger is that once the prototype is complete, you decide to skip the extra work and simply continue work on the prototype with all its hacks, shortcuts and security holes. Don’t do it! Instead, plan you security measures thoroughly from the beginning. Do not simply say: “We’ll run striptags on all output and addslashes on all input”, but set out an abstraction layer which allows you to secure all input and output, not matter where it’s going. Decide on security policies and content rules beforehand, instead of patching your old code when someone breaks your system.

This might seem like a lot of work, and it is! It is, however, also very necessary to prevent all sorts of nasty security breaches. The upside of course is that you do not have to implement these security measures when prototyping. In fact, you SHOULDN’T implement them at this stage. Prototyping is about coming up with something workable quickly to see if it works as intended and according to plan. This does not require excessive security precautions. Just remember to put them in when you begin developing the real system.

Use as few images as possible and merge those you can

Everyone does not have a fast internet connection. In fact, there are still those out there browsing the web on a 56.6kbit modem, and although they are a minority, it should tell us that the excessive use of images and other external media is not exactly taking care of your users. Rather the contrary. That is not to say that you should never use images on a site, in fact images are usually essential to a visually appealing website. The danger is excessive use of media.

A common misconception about external media on web pages is that as long as one uses small images, everything is fine. The truth is that the fewer files the browser has to fetch, the better. Every new request to the server comes with overhead and delay waiting for the server to send its response. Where you can, merge together your images and use CSS sprites to display only the part you wish to show. This saves you from the overhead and increases the overall compression rate of your images.

Don’t make a mess – modulate

When you develop websites, it is all too easy to put everything in one file. When prototyping this makes your development speed much higher, but as your system grows it will soon become slow and unmanageable. Instead, try to split your system in to logically separated units. A good place to start is to implement a MVC oriented framework for your site.

Avoid the lazy fallbacks

If you can’t understand how to do something the first time around, don’t fall back to the easy solutions.

• Tables are for tabular data
• Absolute positioning is for overlay windows
• Frames are an absolute no-no

Instead, try to learn something new, and ask someone who knows more about the problem than you do to help you out.

When asking for help

If you are not already a programmer, do not make the mistake, as many do, of demanding answers or asking for a piece of code instead of advice. There are plenty of skillful people out there willing to help, but they will not help you if:

• You do not ask nicely
• You do not provide source code and other relevant material for them to examine for problems
• You ask for the complete solution
  • Rather than saying “I need a script that does x”, try to make one that does it yourself, and then ask: “I’m trying to do x, and have come up with y. I seem to be stuck because of z. Has anyone got a suggestion about how I might accomplish this?” where y is your source code and z is your problem.

Essential hacks and gotchas:

In the world of web development, there are some gotchas that are very very common, but are not too often explained. I will try to bring up some of those here:

The “overflow: hidden” fix:

Say you have the following code:

<div>
<div style="float: left; height:200px;"></div>
<div style="float:right; height: 300px;"></div>
</div>

How tall would you guess the parent div to be? 200? 300?

It will in fact be 0px tall. The reason is that the browser does not count floated elements when considering the height of the element. This “bug” becomes very apparent if you, for instance, try to position something absolutely a certain distance from the bottom within the parent div, because that element would really be position from the bottom of the tallest NON-FLOAT element within the parent.

There are several fixes for this problem, but the most common (and cleanest) method is simply to put the style “overflow: hidden” or “overflow: auto” on the parent. For some reason the browser then takes the floating children into account and correctly sets the height of the parent.

Absolute positioning

Consider the following code:

<div style="margin: 50px;">
<div style="position: absolute; top:50px; left: 50px;"></div>
</div>

Where would you say the child div would be positioned in relation to the browser viewport? At (100, 100)? You would be wrong. The correct answer is at (50,50). The reason for this is that absolute positioning is considered relative to the first parent element that has been given a “non-flow” position. That is, any element that has its “position” style set to something else than “static”. If no such parent exists, it is positioned in relation to the viewport. A quick fix is simply to set the style declaration “position: relative” on the parent you want to absolutely position the child in relation to. Because relative positioning does in fact not move the element at all unless left and top are specified this means the rest of the page is not affected, and we can get on with our work.

Due note however, that doing this means that ALL absolutely positioned children of the element that receives “position: relative” will be positioned relative to that element. Thus, you cannot have two children elements where one should be relative to the viewport and one relative to the parent.

Full height column backgrounds

A common design layout is one which has multiple vertical columns (often two or three) next to each other. In the designs, these columns almost always are the same length, with the background color extending to the very bottom of the elements. The problem is that whilst this is very much possible in Photoshop, in HTML an element is only as large as its content or as large as you specify it to be. And unfortunately, CSS does not allow you to specify an element to be as tall as another element. Therefore, if you have multiple columns, and at least one of them has content that will vary in height, you will inevitably end up with the columns being different height. If you then set a background color on each of them, you will notice that the background color is only drawn on the element, thereby making it very obvious that the columns are not the same height.

So, how do you make the columns appear to both be as high as the tallest column? Faux columns. In essence, this approach depends on the parent being as tall as the tallest of the children – which it will be unless we’re floating the columns; in which case we can apply the “overflow: hidden” trick. A background image containing all column backgrounds is then applied to the parent and repeated down its entire height. Read the link for full implementation details.

Working with IE

Let’s face it – IE makes our life terrible. With IE8, Microsoft is starting to get it right, but IE7 and particularly IE6 gets a lot of things wrong. Luckily, the fact that they don’t follow the standards also gives us several methods of giving specifically tailored instructions to IE. There are two ways of approaching IE specific hacks – the quick, simple way, and the longer but better way:

The quick fix

Targeting IE6 in CSS: Prefix your selector with “* html “

Targeting IE7 and 6 in CSS: Prefix the attribute with a star (no space between the star and the attribute) – Note that this hack will invalidate your CSS!

The good fix

Use the Internet Explorer only Conditional Comments

Jumping pages

Okay, so you’ve made you site perfect. It is centered, looks beautiful and navigation is smooth as a kitten’s hair. Just one more page to test – the “About us” page with lots of text… As you click the link, your entire page jumps slightly to the left. You try to figure out why, and notice that you now have a scrollbar which means the center of the page has moved, and so, dutifully following your CSS, the browser has moved you page to the new center.

The solution to this annoying problem is to make sure that the scrollbars are always present, but are grayed out when there is nothing below the fold. And how do you do that? Like this:

html {
	height: 100%;
}
body {
	min-height: 100.1%;
	overflow-y: auto;
}

Centering

CSS has many ways of centering, to the confusion and irritation of most developers:

text-align: center – This makes inline elements center horizontally in its parent element

vertical-align: middle – Should be applied to inline element and sometimes centers and element vertically if you’re lucky (press “sometimes” for more details)

margin: 0 auto – Centers a block-level element by setting its left and right margins to the same value. Here, IE of course has to mess up the beauty, and requires the parent to have “text-align: center” in order for it to work. Remember to reset the text-align to left inside the element!

position: absolute – This one requires a bit more explanation. The idea here is that you position an element in the center of another by moving it 50% from the top and 50% from the left, and then move it back by half its dimensions using a negative margin. For example – if you wanted to center an element that is 400×200px within its parent, you would first of all set “position: relative” or similar on the parent (see the Absolute positioning headline further up in this post), and then you would set the following styles on the element itself:

position: absolute;
/* Center vertically */
top: 50%;
margin-top: -200px;
/* Center horizontally */
left: 50%;
margin-left: -100px;

If this does not make sense, read it again.

display: table-cell – This quite new method relies on using CSSs ability to render any element as if it was a table cell, and then using the vertical alignment property of a table cell to center the content. It is especially good for centering text! View it here.

Use a Doctype

A Doctype is not something one puts in simply to make the page validate, it does actually have an effect as well. In the case of some browsers, the determine whether to render the page according to the standard or not based on the presence of a Doctype. Therefore, put it in!

Final words

This non-exhaustive list contains my experiences from web development so far, and will probably be expanded upon by both me and hopefully some of you (use the comment field below!). Use it for what its worth, and avoid the pitfalls that are all too easy to fall into when one is new in the field. If you have any questions or comments regarding these notes, feel free to post your comment below, and I’ll do my best to give you a proper response!

Happy coding!

Recovering data from a Mac drive from Linux

The Mac of a friend of mine crashed the other day – complete harddrive failure. He turned it into the Apple store, and they decided to give him a new disk because they said they couldn’t recover the old one. Somehow, he managed to talk them into leaving him with the old drive so he could try to get at least some of his data back. My friend then came to me, and asked me to have a look at what I could find.

He had already tried to hook the drive into another mac, but it just froze every time he tried to enter a folder on the drive. The same happened when I used my SATA docking station, and attempted to either access or repair the drive through a piece of software called MacDrive. It seemed as though all hope was lost.

I decided to give it another shot from Linux, just to see if I could somehow avoid the corrupt files, and copy only those that could be properly read. Turns out, Linux didn’t even break a sweat when seeing the corrupted harddrive. All I had to do ( using Arch Linux that is ) was this:

1. Attach drive to my SATA dock
2. Mount the drive: “mount /dev/sdg2 /mnt/ext”
   • This assumes the external drive is sdg, check “fdisk -l” to find attached drives
   • Mac-formatted drives have at least two partitions ( one is the boot partition ). Therefore, you must mount partition two ( or whichever is formatted as HFS+ )
   • Because Apple decided to use journaling on their HFS+ drives, these are not writable from Linux. Either live with it, or insert the drive into a Mac, open up the terminal ( usually Applications/Utilities/Terminal ) and run “diskutil disableJournal /dev/disk#” where # is the drive number. Find the drive number by running “diskutil list”. For more info, see: http://castyour.net/node/40
3. Run “cd /mnt/ext/”
4. Navigate to whatever folder you want to copy and run “cp -R <folder> <destination>”
5. The cp command will then dutifully copy all the files it can read properly into your destination folder, and tell you if it can’t read a file. It will automatically skip them.
6. Your files are saved!

Yahoo Pipes

Ever wanted to grab some data from a webpage, or perhaps several, manipulate it in one way or another, and then format access the data through, for instance, JavaScript? Earlier, you would have to write such a script in PHP, ASP, Perl or another scripting language on your web server, and then get the data from that script through JavaScript. This meant you had to run a web server in the first place, and that it would have to support not only fetching data remotely, but also the scripting language you chose to use. And if you didn’t know any scripting languages apart from JavaScript, you would be out of luck.

The engineers over at Yahoo came up with a solution: Yahoo! Pipes. Pipes allows you to set up a series of, well, pipes, for your data to go through, each manipulating it slightly, until you have processed your raw data into what you wanted to extract in the format you want it.

Pipes is an essential tool in today’s world of mash-ups ( that is, sites that do not have any content themselves, but just collect data from other sources, combine and manipulate them, and then publish the processed data ), because it allows you to do most of the work on a remote server ( Yahoo! ), through a simple drag-and-drop GUI ( Pipes ), and fetch the end result using nothing but JavaScript. You can of course access the data using whatever technology you want, since you determine the output format.

Just to give a simple example of what Pipes can do: Say I wanted to create a feed for my users that contained all my posts, except those that contain the word “task”. In order to do this before Pipes, I would have to write up a new PHP script that would fetch all my articles from the database, filter out those with “task” in them, and create a whole new RSS document. Replicating large portions of the code that renders my current feed. Or, I could change my current RSS feed generator so that it accepted URL arguments that allowed filtering by words. All of this would require me both to edit files on my server and make my scripts more complex.

With Yahoo! Pipes, I can simply set my feed as the input source using feed auto-discovery, set up a filter to block items containing “task”, and output the resulting elements as RSS XML. Done!

You can see my pipe for doing this at: http://pipes.yahoo.com/jonhoo/nontaskblog

Oh, and did I mention you can publish pipes…? ;)

PS: If you want to fetch the feed form a WordPress blog, notice that you should not input the feed URL directly, but rather select the source to be “Site Feed”, and then input the URL for the front page of your blog.

RSS and XML

One of the greatest merits of the web, and at the same time, one of its greatest vices, is its decentralization. Data is scattered all over the place; and although this makes for a diverse and ( to a large degree ) uncensored source of information and news, it also means that you have to go to several different sites whenever you wish to see what has happened since you were last online.

Luckily, this problem has a solution – syndication feeds. A syndication feed, which means the supply of material for reuse and integration with other material, is a machine-friendly list of newly updated or created items on a site containing metadata, and usually a piece of the body of the item. One site can have multiple such feeds, each catering for a different category or section.

The advantage of such feeds is that they are machine-readable. This means that you can have a piece of software either online or on your local computer which aggregates data from multiple such feeds, and joins them together into one large list of articles and updates for your viewing pleasure. No longer any need for visiting all those pages, instead, let a feed reader such as Google Reader do the work for you, and then just browse through the list of updates from the pages you subscribe to.

So, how does this work? Well, there are several different feed formats ( unfortunately ), but the most common ones are Atom and RSS. Both rely on XML do describe the data, but with slightly different structure and each their advantages and disadvantages. Today I will be discussing RSS v2 ( there is a v1 as well, but it is currently being abandoned in favor of v2 ).

A RSS v2 looks like this: http://jonhoo.wordpress.com/feed/. Your browser has probably slapped on a pretty interface, but if you right-click the page and press “View Source”, you will see the underlying XML. It will look something like this ( abbreviated for clarity ):

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
        <title>Tech That!</title>
        <link>http://jonhoo.wordpress.com</link>
        <description>The rantings of a mad scientist</description>
        <language>en</language>
        <image>
                <url>http://www.gravatar.com/blavatar/6d568a76e1cf896c928d6ee52b5330f2?s=96</url>
                <title>Tech That!</title>
		<link>http://jonhoo.wordpress.com</link>
	</image>
	<item>
            <title>Portfolio live</title>
            <link>http://jonhoo.wordpress.com/2009/11/13/portfolio-live/</link>
            <pubDate>Thu, 12 Nov 2009 17:21:12 +0000</pubDate>
	    <category><![CDATA[Me]]></category>
	    <category><![CDATA[Tech]]></category>
	    <category><![CDATA[portfolio]]></category>
	    <category><![CDATA[Jon Gjengset]]></category>
	    <category><![CDATA[thesquareplanet.com]]></category>
	    <category><![CDATA[live]]></category>
	    <guid isPermaLink="false">http://jonhoo.wordpress.com/?p=85</guid>
	    <description><![CDATA[Finally, my long-awaited ( on my part at least ) portfolio is live!
                You can see it at http://thesquareplanet.com
                Design and coding entirely by me..]]>
            </description>
        </item>
</channel>
</rss>

This probably looks quite confusing, but let’s take it bit by bit. Firstly, notice the root element of the XML: <rss>. This is compulsory for any RSS feed. Second, notice the version number also has to be included.

Next, we define our first channel. A channel is almost like a folder or a section-container. Everything put in a section is a part of the same category. We then proceed to describe the channel by giving it a title, a link and a description, as well as telling the reader what language the feed is using. All of these ( except language ) are compulsory.

Notice also the <image> preceding the metadata about the channel. This allows the reader to display a feed image to the end-user to easily identify your feed from a list of many. Sort of like a favicon for your feed.

We then proceed to our first item. Again, we specify metadata about that item such as its title, its publication time, a list of categories and a link to the element. WordPress also specified as GUID – a Globally Unique IDentifier. This is a string that should uniquely identify this one item in any collection of URLs.

The last tag in <item> is the <description> tag. Either this or the <title> tag HAS to be present in any valid RSS v2 feed. The <description> tag should contain an excerpt of the item.

And that’s it.. Nothing more is needed to make a full RSS feed, with all its glory and genius. Happy feeding!

For a complete overview of the RSS standard, have a look at http://cyber.law.harvard.edu/rss/rss.html

Portfolio live

Finally, my long-awaited ( on my part at least ) portfolio is live!

You can see it at http://thesquareplanet.com

Design and coding entirely by me.. The coding was not really much of a feat, as it is quite simple, but design has never been my thing, so it was fun to get to stretch my visual creativity legs ( we’ve all got them ).

Drop-in PHP folder gallery

Update 21/11/09: The script now supports thumbnails for video and text, as well as timecodes for video and audio. FFMpeg is needed though..

Have you ever uploaded a bunch of images to a new folder at your webserver to share them with others, and ended up with just a listing of clickable filenames? No previews or thumbs of anything..

Well, I’ve found myself in that position too many times, and decided to create a standalone drop-in PHP gallery file that pulls thumbs from images, videos and text files and displays them in an easily scannable format. The thought behind it was that it should be a single PHP file that could be dropped in the directory and left there without any more work.

The file is not complete, but it does work with images so far, and video and text support is half-way there.

You can see the alpha version here: http://jon.thesquareplanet.com/index.gallery.phps

To see it in action, see: http://jon.thesquareplanet.com/bond/toga/

Hot-swapping drives in Windows 7

I have recently bought two external SATA docking stations – one internal ( i.e. it fits in a 5.25″ bay, and loads the drive from the outside like a large floppy ), and the other one completely external and connected through eSATA. For the first couple of weeks, I thought hot-swapping was not possible with these, and kept rebooting if I wanted to swap out a drive, however one day I came across a setting in BIOS describing how SATA drives should be treated. It was set to “IDE compatible”.. The other options were “Enhanced” and “AHCI”. I tried googling this, and soon found that AHCI is actually a technology that enables plenty of the cool features of the SATA technology – most notably hot-swapping!

I enabled AHCI, booted up the computer, and Windows 7 presented me with a BSOD… Again, Google was my friend, and I found several other people who got the same problem when enabling AHCI after install. It seems as though Windows 7 checks for AHCI when installed, and determines then whether to load the AHCI drivers or not.. It then never checks again…. Smart…

Luckily, there is a solution.. First you have to get back into Windows my resetting the SATA drives to “IDE compatible” mode. Next, open up the registry explorer, and follow this guide: http://support.microsoft.com/kb/922976. If it is already set to 0, set it to 1, then to 0 again, and reboot. Now, set the drives to AHCI in BIOS, and reboot again. Hopefully your Windows should start up without a bluescreen.

Now, all your SATA drives will appear  when you use the “Safely remove device” icon near the clock in the bottom right corner. If you choose to remove a drive, you can eject it from there and then take it out and put in a new one. This is where the problems start to arrive though. Sometimes, this approach works without a problem, but sometimes Windows simply goes silent, and acts as though nothing has been connected at all. Other times, it tells you that it has found partitions, but that they are in RAW format, and have to be reformatted!

After living with this for a couple of days, I decided that there had to be a more stable way of doing this, and that was when I came across HotSwap!. This piece of software is made for managing hot-swap drives in Windows, and once installed, allows you not only to scan for new drives ( and load them properly! ), but also to safely remove them AFTER doing a spin-down.

After running this stand-alone .exe, you can set it to autostart with Windows. After that, whenever you want to swap a drive, just right-click the hotswap icon at the bottom right near your clock, safely remove the device you want to swap, exchange the drives and choose “Scan for changes” in the HotSwap! menu and up comes your drive!

Happy hot-swapping!