Saturday, June 15, 2013

Drupal behind heavy doors

We just upgraded the largest daily newspaper in Israel, "Israel Hayom" (, to the Drupal platform.
Drupal is a great CMS which suits the data structure very well and allows us, the developers, to hand off much of the controls to the editors and site administrators, with lots of very cool features and great user experience. On the contrary, when it's comes to delivering pages and content to the end-users, Drupal may not be the best solution. Direct access to Drupal can result in a very weak performance while every content request can become an obstacle.

Our team approach was to put Drupal “Behind heavy doors”. Which means, no content request will be granted a direct access to Drupal, but ours. We’ve divided our solutions into two parts: one solution for the static content and one solution for the interactive, dynamic content.

Most of the news content on the website today is presented as static, and the best solution for delivering static content is done by caching. All the static content as articles, opinions, etc. is being cached as HTML pages. Our solution for improving static web pages performance with Drupal is divided into two layers:
 The first layer which the end-users come across is an Akamai CDN. Akamai is a few thousand servers spread globally and caches HTML pages by URL path, each page is assigned a high TTL (Time To Live) of one week. This process requires a lot of caching but if a new page is delivered or the cache has cleared, the request will pass on to the next layer of the solution.
 The second layer is a Ngnix used as a reverse proxy server which saves all the static pages as HTML and never clears the cache. Each submission (Insert or update) of any content on the website by an editor will automatically be “pushed" to the server. The caching action is reduced by only occurring on content submissions and not on content requests. This is what makes the Drupal server impermeable (almost :-)) for end-users requests.
To find out how we solved immediate content updates on Akamai, please click here.

The website is made out of a variety of both Static and Dynamic pages.
Any list of content used in combination with user selection options, as filtering, sorting, search module or even just using a pager, will automatically become an interactive page with dynamic content, which does not use cache system.
To resolve the fact that we cannot cache those pages as HTML, we built two corresponding systems:
 The first one is a NodeJS / MongoDB server, which synchronizes all the content. On each submission we store a content-object with the relevant data on the MongoDB, ready to be retrieved at any time. On the other side we structure a data API for querying the data-objects, by passing fields, filters, sorts, etc. (This API is also serves all of our external providers like mobile devices, smartTV, etc.). At this point, we turned all the dynamic pages in the website (These are usually built with Views module), to a client-side JS rendering mechanism with AJAX and a template engine (aka Backbone.js + Mustache) that interact with the end-users, and dynamically querying the API which has super fast response time. (
 The second system is an Apachesolr server which holds similar objects and ready to query any searches initiated by the end-users, from the website, mobile device or smartTV.

As you can see, direct access for users request is not occurring to the Drupal server, the routing system or database. All of the complex processes are fully controlled, maintained and performed at the background while they are out of reach or Behind heavy doors”.

Friday, April 12, 2013

Pushing live updates to CDN (Akamai)

If you ever tried to live-updated your site which is hosted on a CDN, you realize it's take time for the update to take affect.
On big networks like Akamai it's can take really long minutes. Clearing the cache on all the network servers, can take ~40 min. Sometimes you need a live update in a matter of one minute, top (or even less). For example, on newspaper website, More than that, and your readers are gone.
One approach we conceived and found useful is working like this:

Using the advantage which CDN consider query-strings as new page and save it to the cache while the origin server consider it as the same page. 

On the server side:
When an editor is update an article it send a request to node.js server which saves a record to a MongoDB (or Redis) with the article ID, current version (increase number) and a timestamp.

On the client side:
On the article load there an AJAX requesting the node.js server, who querying to our MongoDB with it's article ID asking for version No. If any, there a content switch occurs by requesting for the new version using query-string e.g. Now there two scenarios, if this path is already cached its just grabs it from there. If not, the request hit the origin server which ignore the query-string and get the new, updated article. 

With this system we were be able to updated live articles in a real-time, and still manage an aggressive chach with an high TTL (Time To Live) of approximately a week long. While most of the articles are never changing or updated.

Thursday, March 14, 2013

Git working process with QA

Lately we switched our version control (GIT) workflow to better control our QA, and each phase of it, in our regular development / code-deployment process.


  1. Version control system (GIT).
  2. Task management (redmine).
  3. Remote server assignments (Jenkins).

Task Types:

  1. Hotfix - Short term task (no more than a few hours), with a single developer.
  2. Release - Long term task (usually a few tasks with the same theme), with multi developers. Or just one of the two.


  1. Master branch - is the stable branch, which acts as feeder to all of the developers’ feature branches. Updated only by the QA guy with approved production tags. Master is updated with the current production tag.
  2. Feature branches for release-type tasks - remote gets updated.
  3. Feature branches for hot fixes - remote does not get updated, local branch only.


  1. Local - developers’ local environments.
  2. Development (multi) - for pre QA testing and sharing. Updated using GIT tags.
  3. Stage/QA (multi) - updated only by QA with GIT tags.
  4. Production - updated only by QA  with approved GIT tags.

Developer rules:

  1. All work is done in feature branches.
  2. Never update Master branch.
  3. Master branch will be updated only by QA from approved GIT tag.
  4. Feature branches are updated (merge) frequently from Master branch.
  5. Done tasks will be committed and tagged, e.g. q-1.1, and pushed to remote.
  6. Feature branch gets deleted once it’s no longer needed.

QA Rules:

  1. Tags are tested one at a time. A new tag gets tested after the previous one has finished the cycle, and has been updated on Master branch:
  2. The new tag is checked out and merged-to from Master (with the latest approved features in it).
  3. Stage environments are updated with testing tag, e.g. r-1.1, which contains Master branch (equal to prod) with the new feature.
  4. Approved tags are re-tagged, e.g. p-1.1, for production update.
  5. Approved tags are merged into Master, for feeding all developers and next tags.


for developers

    1. Create a new branch updated and equal to Master.
    2. If it’s a release task, push the branch to remote.
    3. Perform the task, in the best way you can, and commit.
    4. Merge from Master (git pull to update local Master from remote Master, and then git merge to your branch from local master).
    5. Tag the current state for QA testing, e.g. q-1.1.
    6. Push the tag to remote and assign the task as QA in the issue tracking system.

for QA

    1. Fetch the tag and checkout to be on it.
    2. Merge from Master branch, to get the latest approved features and updates.
    3. Create a new tag for a stage test, e.g. r-1.1, which contains Master branch (equal to prod) with the new feature.
    4. For approved tags, re-tag for production, e.g. p-1.1.
    5. Update production with the new tag and test the environment.
    6. Merge to Master the latest tag and push Master to remote.
    7. Close the task in the issue tracking system.

In GIT language:

For QA guy:

Get all tags
git fetch --tags
Move to the tag
git checkout q-1.1
Update from Master
git merge master
Create new tag for testing
git tag r-1.2
Push the tag to remote
git push origin r-1.2 / git push --tags
Test on stage and approved

Move to the approved tag
git checkout r-1.2
Create new tag for update production
git tag p-1.1
Push the prod tag to remote
git push origin p-1.2
Update Prod test and approve

Move to Master branch
git checkout master
Merge the approved prod tag to Master
git merge p-1.1
Push updated Master branch to remote
git push

For developers:

Create new feature branch up from master
git checkout -b f1234
Commit changes
git commit -am”message”
Update local master from remote
git pull origin master:master
Merge feature branch from master
git merge master
Make sure nothing break

Create tag for QA testing
git tag q-1.1 -am”message”
Push the tag to remote and QA
git push origin q-1.1 OR git push --tags (for all)
Coming back to master
git checkout master
Delete feature branch
git branch -D f1234
Start over

* Sometimes it’s make sense to stay on the local dev branch for few task and reset the branch to master between each task using git reset --hard master

Git Workflow Diagram: