It’s hard to find popular web publishers without a mobile strategy today. When asked about mobile, you’ll often hear “We want to do an iPhone app”, “We’re building a mobile site”, “It’s coming soon”, etc.

The top-tier publishers are the few that actually deliver on these strategies. Tens of thousands of dollars later, they arrive to a good mobile presence. The mid-tier publishers – sites with hundreds of thousand unique visitors a month – usually end-up ignoring the problem. Not being able to pay the exorbitant fees of boutique mobile agencies and not willing to resort to a simple site builder or an RSS feed, the mid-tier does not publish to mobile. Most websites covering mobile are not mobile-optimized, which is a sign of a bigger problem.

Mobify is the vehicle for empowering a mobile strategy for great websites without huge mobile budgets. It creates a live, evolving mobile view that is never detached or segregated from your main site. The power to design this view is in your hands. We hope you enjoy it =)

Continue Reading →

Mobify.me acts as a browser and a caching proxy, with all the benefits and limitations that entails. One thing we need to do a lot of is fetching URL’s. We need to cope with every situation a browser may encounter including sites with different types of state, cookies, POST and GET requests, redirects and error codes.

Getting what you want from the web isn’t as easy as you might like. We started using httplib2. This is an all python library that supported all the basic functionality we needed. We found httplib2’s pure python implementation appealing and it wraps the tried and trusted httplib but adds easy caching, persistence, redirects, cookies and compression among other things. Overall, we were pretty happy with it, but it had one limitation: it uses pysocks for connecting to a proxy. We use a squid cache http proxy on our back end to speed up page retrieval, but also to throttle and manage connections. Unfortunately, pysocks only currently supports the CONNECT tunneling method of proxying. Squid permits tunneling, but does not apply any of it’s limits (such as max body response size) to CONNECT tunneled responses.

We’d been looking at using pycurl for a while. Since it wraps to libcurl, a native library, it has a reputation for performance. It is a very actively maintained project and is probably the most developed URL fetching mechanism living outside of a web browser. First, we tested relative performance of pycurl vs httplib2. Our results: both are very fast, but pycurl is about a factor of 4 faster. If you aren’t doing hundreds of requests a minute, performance is not a good reason to choose pycurl. But for us, the performance and the ability to use an HTTP based proxy was critical. Read on for more thoughts on the switch and tips for using pycurl.

(more…)

Continue Reading →