Drupal Planet

Increasingly our web pages are being bookmarked on iOS devices.  This is a good thing, however we had traditionally been using long titles on our pages and these were clumsy for end users when trying to store bookmarks.  Like 90,000 other websites we have chosen Page Title to solve this issue.  It works well with Tokens and Views and is probably one of those modules you should just install.

Here’s your 1:30 video http://www.youtube.com/watch?v=DMvXVJ2mbGQ

and a couple of screen shots for the really curious types.  It’s a simple module to install and use. Made by several of Drupal giants robertDouglass, JohnAlbin, and nicholasThompson.

Turn on page titles in the content type area

Enjoy refreshing page titles independent of your content title

Packt has contacted me in order to let me give away books to three participants. The details of what you have to do can be found below.

a cat from flickr

Developing for Drupal can be exciting, but without a proper approach it can also get messy.

Drupal is maturing and its developer community is growing and becoming more diverse as people from different software engineering backgrounds join. In addition, the number of modules produced is increasing, as is their complexity, especially as Drupal is required to integrate with external services and legacy systems. In the face of this it is crucial that, as a community, we ensure that the experience of coding for Drupal and the quality of the resulting modules continues to improve. If we neglect these issues, we risk negatively impacting Drupal by having poor quality modules and fewer developers wanting to join and dedicate time to the project.

While, as a community, we have specifically focused on improving the Drupal site builder and Drupal designer experience, we should ensure that we don’t forget our main constituency: Drupal developers themselves. I am not talking about core developers or leading Drupal module developers. Rather, I’m referring to the thousands of Drupal developers who day-in and day-out build and expand Drupal modules whether custom or not - and how we as a community provide tools to allow them to do this work effectively.

Furthermore, the types of tools I am referring to are not just things like more IDE plug-ins, more APIs, more documentation, more snippets and recipes; although all of these are part of the solution, we already have plenty of them.

What is missing are wider-ranging conceptual tools that describe a rational and systematic approach to choosing and gluing together all the different systems and APIs that Drupal makes available in a way that can work well, can be shared across team members, and will naturally leads to better quality code. Such tools ultimately improve the developer experience as they reduce confusion, improve confidence in the choices made, and reduce the time it takes to develop. These tools are also more scalable, since they deal with overarching concerns and should survive specific Drupal versions.

In particular, I am referring to the need for methodologies and patterns:

Methodologies: guidelines, principles and methods that we employ to go from problem definition to solution.

Patterns: proven reusable solutions to common problems.

In the following paragraphs I will touch on each and give some examples of the types of things that we as a community could be discussing and presenting to help improve module development. I will leave more specific examples and a more in-depth analysis of the issues to subsequent blog posts.

Methodologies

A methodology is a process that accompanies us from problem definition to solution and provides tools that will allow us to do this in a consistent way. Underpinning a methodology are guidelines or principles that help us determine whether an approach is right or wrong.

In the next couple of paragraphs I will give some examples of what could be valid principles to adhere to and what processes we could suggest.

Here are three sensible principles to adhere to:

Separation of concerns: are we performing actions at the right level? Is our module interacting with Drupal at the right point and through the right systems in order to do things such as access control, theming, interaction with the database, etc.?

Decoupling: is our system flexible enough? Can different components be swapped out easily? Drupal is a mixture of OO and procedural code, so what are the best strategies for building decoupled systems in this rather unorthodox environment?

Consistency: whatever approach we follow, it is important that we are consistent in how we name things, structure the location of files, and describe what we are doing. This makes it easier to share across teams.

And here are some of the processes a methodology can introduce:

1. Always describe a problem in abstract terms first and not in "drupalese." This ensures that you have a clear definition that can be shared across a team and with a client. Consider the following two statements:

A hotel owner needs to be able to display a list of available rooms with their associated descriptions given an arrival and departure date.
Vs.
I need to get all bookable unit entities and attach a field entity reference to them pointing to Room Description nodes that I can then render in Rooms view mode.

The latter makes the mistake of describing the problem in terms of a possible solution. It’s also an example of how Drupal developers all too often liberally referring to nodes, blocks, panels and views instead of the things these tools actually provide (pages, components of a page, the structure of a page, and lists of things). The former, however, describes the problem in abstract terms, making absolutely no assumptions about how it could be solved, while using clear and precise language.

2. Always describe your architecture in generic terms first, and then in specific terms. Talk about storage, data structure, and interaction with other services without worrying about how that would be implemented in Drupal. Then translate that generic architecture into one that uses specific Drupal systems. This enables you to consider alternatives and once more avoids any assumptions about what a good solution would be.

Patterns

Patterns are a well-known concept in software engineering and probably one of the most effective tools we have to educate developers about good practices in developing software.

Within the Drupal world I see the potential of identifying three different types of patterns that we can describe in generic terms, and accompanied with specific examples from modules that employ them so that developers can quickly learn how to use them.

OO Patterns: Classical OO patterns are increasingly making their way into Drupal modules (a good example being the Entity API module). We should pull these out and explain how they can be used across different modules and situations.

Drupal Patterns: There are certain patterns that seem to be Drupal-specific. An example would be the separation of a module's user interface and functionality in two different modules—such as the Views and Views UI module. We should identify more sensible solutions like these and explain how they can be easily implemented in any module.

Service Patterns: Finally, some patterns are best described in terms of web service interactions and are most suitable to describe integrations with external services and Drupal.

Next steps

What I am trying to achieve here is not to formalize the Drupal module software development process and place a straitjacket around what a developer can do and how they should do it. Methodologies and patterns often run these risks, but they can be easily avoided by ensuring that the approach is light and flexible and focuses on underlying principles rather than complicated processes.

Moving forward I would like to: focus on the following goals:

1. Document more patterns from the Drupal world and share those for comment. This is something that I started doing last year by looking at entities specifically. During a Drupalcon London presentation I talked about examples of how entities are used across different modules in order to inform how they can be used in your own modules. And just a couple of days ago at Barcelona Drupal Developer Days, I talked about the issues mentioned in this blog post, although in a more tongue-in-cheek way ;-).

2. Start working on a very lightweight methodology that we can use internally at Bluespark.

3. Identify other things such as module file naming, file structure, etc., that we could standardize within the wider community and which are not covered by the Drupal code style.

If you find these problems interesting, please join in and post your suggestions about how to improve the developer experience in terms of methodology and patterns within the Drupal world.

Tags: DrupalDrupal Planet

Project Review Wednesday Explained

Many people don't know anything about the project review process on Drupal.org. Anyone can post code to Drupal.org, but before sharing it with others in downloadable releases, new contributors must have a project reviewed by volunteers, who check for basic understanding of how Drupal works, in code, community, and especially security. Because there are too few volunteers doing these reviews, there's a long backlog. Waiting isn't a great introduction to the Drupal contributor community. So in an effort to improve this, we're starting a series of weekly posts highlighting some of the awesome projects currently in this queue, awaiting your review.

There are currently 109 new Drupal contributors awaiting review of their first project. This is a great place to contribute to the community and learn about interesting upcoming projects, for example...

Module: Basecamp

What's it Do?

The Basecamp module uses the Basecamp API to integrate the widely-used project management system with Drupal. It syncs Drupal and Basecamp accounts, displays tasks already in Basecamp, and allows creating new tasks.

Potential Uses

This project should be useful to anyone using both Drupal and Basecamp, as we do at Aten. It could also be the basis of something like Open Atrium Basecamp Import for transitioning between the two.

Look Useful? Review It!

If this sounds like something you'd like to see readily available on Drupal.org, you should review it and help make that happen.

Review It

Pro Tip: If you've never reviewed a project application before, you can find instructions for reviewers on Drupal.org and the Code Review group is happy to help more people get involved.

Drupalcamp or Regional Summit Organizers:  Susan MacPhee dcmistry

D4D. The 4th annual drupal design camp boston, the drupal community's first camp dedicated to design, takes place on sat and sun july 14-15 at the mit stata center. each year, 315+ drupalists gather in cambridge to discuss ideas and take part in interactive sessions on the future of drupal, focusing on everything from usability to technology from a design perspective.

We are thrilled to announce back-to-back awesome keynotes this year, Angela "webchick" Byron with a talk on D8 and Jared Ponchot, Creative Director of Lullabot.

Join us!

http://boston2012.design4drupal.org/

We really need sponsors too!

email us at [email protected]

• Design 4 Drupal

I was thrilled to read the story at Mobile Drupal about how ConsumerSearch is using Pronto.js to expose their huge Drupal content to their mobile application.

Pronto.js is designed to be a high performance asynchronous application framework that makes it simple to chain together components to build sophisticated application logic. It's the JS equivalent of the PHP Fortissimo framework.

read more

MongoDB's query language is good at extracting whole documents or whole elements of a document, but on its own it can't pull specific items from deeply embedded arrays, or calculate relationships between data points, or calculate aggregates. To do that, MongoDB uses an implementation of the MapReduce methodology to iterate over the dataset and extract the desired data points. Unlike SQL joins in relational databases, which essentially create a massive combined dataset and then extract pieces of it, MapReduce iterates over each document in the set, "reducing" the data piecemeal to the desired results. The name was popularized by Google, which needed to scale beyond SQL to index the web. Imagine trying to build the data structure for Facebook, with near-instantaneous calculation of the significance of every friend's friend's friend's posts, with SQL, and you see why MapReduce makes sense.

I've been using MongoDB for two years, but only in the last few months starting using MapReduce heavily. MongoDB is also introducing a new Aggregation framework in 2.1 that is supposed to simplify many operations that previously needed MapReduce. However, the latest stable release as of this writing is still 2.0.6, so Aggregation isn't officially ready for prime time (and I haven't used it yet).

This post is not meant to substitute the copious documentation and examples you can find across the web. After reading those, it still took me some time to wrap my head around the concepts, so I want to try to explain those as I came to understand them.

The Steps

A MapReduce operation consists of a map, a reduce, and optionally a finalize function. Key to understanding MapReduce is understanding what each of these functions iterates over.

Map

First, map runs for every document retrieved in the initial query passed to the operation. If you have 1000 documents and pass an empty query object, it will run 1000 times.

Inside your map function, you emit a key-value pair, where the key is whatever you want to group by (_id, author, category, etc), and the value contains whatever pieces of the document you want to pass along. The function doesn't return anything, because you can emit multiple key-values per map, but a function can only return 1 result.

The purpose of map is to extract small pieces of data from each document. For example, if you're counting articles per author, you could emit the author as the key and the number 1 as the value, to be summed in the next step.

Reduce

The reduce function then receives each of these key-value(s) pairs, for each key emitted from map, with the values in an array. Its purpose is to reduce multiple values-per-key to a single value-per-key. At the end of each iteration of your reduce function, you return (not emit this time) a single variable.

The number of times reduce runs for a given operation isn't easy to predict. (I asked about it on Stack Overflow and the consensus so far is, there's no simple formula.) Essentially reduce runs as many times as it needs to, until each key appears only once. If you emit each key only once, reduce never runs. If you emit most keys once but one special key twice, reduce will run once, getting (special key, [ value, value ]).

A rule of thumb with reduce is that the returned value's structure has to be the same as the structure emitted from map. If you emit an object as the value from map, every key in that object has to be present in the object returned from reduce, and vice-versa. If you return an integer from map, return an integer from reduce, and so on. The basic reason is that (as noted above), reduce shouldn't be necessary if a key only appears once. The results of an entire map-reduce operation, run back through the same operation, should return the same results (that way huge operations can be sharded and map/reduced many times). And the output of any given reduce function, plugged back into reduce (as a single-item array), needs to return the same value as went in. (In CS lingo, reduce has to be idempotent. The documentation explains this in more technical detail.)

Here's a simple JS test, using Node.js' assertion API, to verify this. To use it, have your mapReduce operation export their methods for a separate test script to import and test:

// this should export the map, reduce, [finalize] functions passed to MongoDB.
var mr = require('./mapreduce-query');
 
// override emit() to capture locally
var emitted = [];
    emit = function(key, val) {
      emitted.push({key:key, value:val});
    };
 
// reduce input should be same as output for a single object
// dummyItems can be fake or loaded from DB
mr.map.call(dummyItems[0]);
 
var reduceRes = mr.reduce(emitted[0].key, [ emitted[0].value ]);
assert.deepEqual(reduceRes, emitted[0].value, 'reduce is idempotent');

A simple MapReduce example is to count the number of posts per author. So in map you could emit('author name', 1) for each document, then in reduce loop over each value and add it to a total. Make sure reduce is adding the actual number in the value, not just 1, because that won't be idempotent. Similarly, you can't just return values.length and assume each value represents 1 document.

Finalize

Now you have a single reduced value per key, which get run through the finalize function once per key.

To understand finalize, consider that this is essentially the same as not having a finalize function at all:

var finalize = function(key, value) {
  return value;
}

finalize is not necessary in every MapReduce operation, but it's very useful, for example, for calculating averages. You can't calculate the average in reduce because it can run multiple times per key, so each iteration doesn't have enough data to calculate with.

The final results returned from the operation will have one value per key, as returned from finalize if it exists, or from reduce if finalize doesn't exist.

MapReduce in PHP and Drupal

The MongoDB library for PHP does not include any special functions for MapReduce. They can be run simply as a generic command, but that takes a lot of code. I found a MongoDB-MapReduce-PHP library on Github which makes it easier. It works, but hasn't been updated in two years, so I forked the library and created my own version with what I think are some improvements.

The original library by infynyxx created an abstract class XMongoCollection that was meant to be sub-classed for every collection. I found it more useful to make XMongoCollection directly instantiable, as an extended replacement for the basic MongoCollection class. I added a mapReduceData method which returns the data from the MapReduce operation. For my Drupal application, I added a mapReduceDrupal method which wraps the results and error handling in Drupal API functions.

I could then load every collection with XMongoCollection and run mapReduce operations on it directly, like any other query. Note that the actual functions passed to MongoDB are still written in Javascript. For example:

// (this should be statically cached in a separate function)
$mongo = new Mongo($server_name);      // connection
$mongodb = $mongo->selectDB($db_name); // MongoDB instance
 
// use the new XMongoCollection class. make it available with an __autoloader.
$collection = new XMongoCollection($mongodb, $collection_name);
 
$map = <<<MAP
  function() {
    // doc is 'this'
    emit(this.category, 1);
  }
MAP;
 
$reduce = <<<REDUCE
  function(key, vals) {
    // have `variable` here passed in `setScope`
    return something;
  }
REDUCE;
 
$mr = new MongoMapReduce($map, $reduce, array( /* limit initial document set with a query here */ ));
 
// optionally pass variables to the functions. (e.g. to apply user-specified filters)
$mr->setScope(array('variable' => $variable));
 
// 2nd param becomes the temporary collection name, so tmp_mapreduce_example. 
// (This is a little messy and could be improved. Stated limitation of v1.8+ not supporting "inline" results is not entirely clear.)
// 3rd param is $collapse_value, see code
$result = $collection->mapReduceData($mr, 'example', FALSE);

MapReduce in Node.js

The MongoDB-Native driver for Node.js, now an official 10Gen-sponsored project, includes a collection.mapReduce() method. The syntax is like this:

 
var db = new mongodb.Db(dbName, new mongodb.Server(mongoHost, mongoPort, {}));
db.open(function(error, dbClient) {
  if (error) throw error;  
  dbClient.collection(collectionName, function(err, collection) {
    collection.mapReduce(map, reduce, { 
        out : { inline : 1 },
        query: { ... },     // limit the initial set (optional)
        finalize: finalize,  // function (optional)
        verbose: true        // include stats
      },
      function(error, results, stats) {   // stats provided by verbose
        // ...
      }
    });
  });
});

It's mostly similar to the command-line syntax, except in the CLI, the results are returned from the mapReduce function, while in Node.js they are passed (asynchronously) to the callback.

MapReduce in Mongoose

Mongoose is a modeling layer on top of the MongoDB-native Node.js driver, and in the latest 2.x release does not have its own support for MapReduce. (It's supposed to be coming in 3.x.) But the underlying collection is still available:

var db = mongoose.connect('mongodb://dbHost/dbName');
// (db.connection.db is the native MongoDB driver)
 
// build a model (`Book` is a schema object)
// model is called 'Book' but collection is 'books'
mongoose.model('Book', Book, 'books');
 
...
 
var Book = db.model('Book');
Book.collection.mapReduce(...);

(I actually think this is a case of Mongoose being better without its own abstraction on top of the existing driver, so I hope the new release doesn't make it more complex.)

In sum

I initially found MapReduce very confusing, so hopefully this helps clarify rather than increase the confusion. Please write in the comments below if I've misstated or mixed up anything above.

Today we're going to look at a use of Drupal that is a bit outside the norm. Instead of the usual business website, corporate blog, or news magazine site, let's look at a building a presentation tool. How or why would you use such a thing? Well, in a world where the Powerpoint or document-based presentations and proposals are rampant, sometimes you need to do something to stand out a bit.

read more

Last week, we announced that we had put together documentation and a demo site for Julio, our distribution for schools, school districts, and academic departments for K-12 and higher education.

As we have been building school web sites over the years, a common feature request we received was: "I want people to be able to put content in one place on the web site, but only that place." Translated, this meant that they wanted to decentralize control of the web site, and allow people freedom within the areas where they are responsible.

So, for example, the Football coach can put anything she wants into the Football team section of the web site, but she cannot put anything into the any other place.

As Drupal has evolved over the years (for us, starting in 4.5) we solved this in many different ways. However, starting midway through D5, and with some consistency in D6, we began using Organic Groups as the tool to address this functional requirement. In D5 and D6, this involved overriding many OG features on a pretty regular basis, as the default settings provided options within the user interface (UI) that ranged from distracting to scary, depending on the end user and their privileges within the site.

For Drupal 7, Organic Groups was rewritten from the ground up. To understate things, this was an enormous undertaking. The lead developer on this is Amitai Burstein from Gizra, and on this rewrite, he nailed it. The rewrite makes smart use of entities in D7, and, last November, Amitai made the key decision to simplify Organic Groups by deprecating the OG Group entity. This change has many benefits, and one of the more immediately tangible benefits is that the views integration is now much cleaner, which simplifies the work of site builders. We had been working with OG in D7 for about a year when Amitai announced his proposed changes, and the direction he was talking about dovetailed pretty cleanly with our experiences up to that point. It's also worth noting that Amitai's approach to maintaining OG is pretty incredible; the work required for the initial rewrite of Organic Groups for Drupal 7 was considerable, and the rewrite for the 2.x branch was no small feat either. It would have been easy to stick with the original approach in the 7.x-1.x branch, but Amitai made the call to simplify Organic Groups to make it easier to use, and that's a great choice for all of us in the community.

In addition to sponsoring some of the original development, we have tried to be pretty active in the OG issue queue, helping with patches and testing as part of our ongoing work. We have been working with the 2.x branch of OG since late January, 2012, and it has been incredibly useful. In D5 and D6, delivering a site built on top of OG required putting a significant amount of work into UI tweaks. The core functionality of OG was (and remains) incredibly flexible, but the rewrite simplifies the process of delivering a site that has powerful and extensible community sections while being easy to use. From a site maintainer/builder perspective, one of the main improvements in the D7 version of OG is the ability to provide rights on a per-group basis.

Ongoing development and improvements are taking place in the 2.x branch; if you have been waiting to test this out, well, what are you waiting for? Grab a copy, and start testing, now! Yes now! What are you waiting for?

For more information and background on Organic Groups, check out Amitai going over some of the details from his DrupalCon Denver session: OG7 - Pride and Prejudice.

Image Credit: "grilled sugar snap peas" taken by woodleywonderworks, published under an Attribution Non-Commercial No Derivatives license.

Clients often require that a site pass a validation tool. I'll detail some tips and tricks for getting a Drupal 7 site to validate with HTML5 and RDFa.

ConsumerSearch (part of About.com and the New York Times) is hiring! We are looking for excellent software engineers who want to work in a technology agnostic environment!

In February 2012 our team at ConsumerSearch launched the ConsumerSearch Reviews iOS app. This handsome app helps you during your product purchase by providing extensive product reviews, comparisons, pricing information and online and local store product availability. Try it out during your next purchase -- you'll be surprised how much time you'll save in researching the best products.

Searching for products is possible by keyword search or barcode scanner. We are very proud of our first app that has already gone through several updates and has received positive user reviews.

cs-iphone-screen-1

read more

We've tried a lot of project management systems over the years. In some way, they have always seemed lacking, confusing or just a pain in the rear end. If they had good tools for project managers, they were confusing to developers. If they were useful for developers, designers complained about the eye-sores. No one system ever seemed to satisfy the team.

We recently started using GitHub for project management after the developers started raving about how much they loved it for managing code. To our surprise, GitHub has proven a solid option for project management. Our designers have even started using it for some of their projects, which I think says something about GitHub's aesthetics. With a little bit of something for each role, GitHub is starting to come out on top as the tool of choice for hosting code, managing projects, and facilitating project communication.

Content management coupled with content strategies are the perfect 1-2 punch for the ultimate results oriented website.

The biggest revolution in the history of marketing is in full swing. If you are in business and you are on the web, you are in the publishing business. You need to publish great content that attracts audiences and gets them excited about your brand – and, oh yeah, gets them telling their friends.

read more

Planning to set-up new development environment, but haven't yet quite decided between MAMP and installing Apache and MySQL locally?
There is also 3rd option, run LAMP stack virtuallized locally.

Benefits of running your XAMP (apache, mysql & php) stack in virtualbox has many benefits.
- Like having same environment as in production.
- All libraries are not (easily) available to all platforms.
- Freeing resources is easy, just shutdown your virtualmachine.

Just this past week, we had a full week of sprinting on Drupal 8 Multilingual issues on ocassion of the Drupal Dev Days over the weekend. Citilab, the venue for the event was generous to let us use the space for the whole week, and four fantastic financial sponsors, dotProjects.be, Cando Image, Open8 and Acquia made it possible for the sprinters to have a worry-free week. Their sponsorship let us have our coffee, soft drinks, lunches, ice cream, fruits and vegetable snacks and even a very nice dinner downtown covered, so we can just focus on what we are good at: solving multilingual issues for Drupal 8. We absolutely grateful for their support.

We have had about ten people starting off the week sprinting which grew up to over 20 by Friday. Key people like Internationalization module maintainers Jose Reyero (reyero) and Florian Weber (webflo), Localization update maintainer Erik Stielstra (Sutharsan), Entity API lead Wolfgang Ziegler (fago), CMI initiative lead Greg Dunlap (heyrocker), core generalist Daniel Kudwien (sun), core committer Angie Byron (webchick) were around, which really helped us make great advances. Amazee Labs really put the multi in the sprint by sending not one but two great developers, Micheal Schmid (Schnitzel) and Vasi Chindris (vasi1186). We've had long time hard workers Peter Droogmans (attiks), Christian López (penyaskito) and Clemens Tolboom (Clemens.Tolboom, remote) in attendace and even Francesco Placella (plach, remote) could contribute all the way from Venice. We also had people helping us test our new user interfaces and figure out problems, including Lowell Montgomery (LoMo). Various people were sprinting focused on Entity API, including but not limited to Sebastian Siemssen (fubhy), Sascha Grossenbacher (berdir), Miro Dietiker (miro_dietiker) and Frederic Marand (fgm, OSINet). Thanks again everybody for coming and participating! (And I'm sorry if I forgot to mention your name).

We have made great advances in all four areas of the Drupal 8 Multilingual Initiative (base language, interface translation, content language and config language).

Key results already landed in Drupal 8

All language globals are now gone

The process of removing language globals is now complete. The Dependency Injection introduction that was started with the language negotiation values in http://drupal.org/node/1497230 is now the sole way to access negotiated language values for the page. See the change notice at http://drupal.org/node/1539454.

Language class is now used universally

The Language class that was introduced with the Dependency Injection process is now used universally with language_list(), language_load(), language_default(), etc. This means you can use type hinting for parameters that are supposed to be language objects. Further work to make the Language class full featured is being discussed at http://drupal.org/node/1512424

Introduced locked languages, more special languages exposed

We broke down the Drupal 7 'Language neutral' language to 'Language not applicable' and 'Language not specified' earlier and added 'Multiple' as language constants, but we did not have a UI to actually assign these languages to things. Now, you have these languages show up on your language list with explanation of their roles and you can reorder them and apply them to content, etc. See http://drupal.org/node/1471432 for the issue introducing this change.

Locale module is now purely for interface translation

In Drupal 6 and 7, Locale module has gotten fatter and fatter with code to solve language problems on behalf of other modules. Most of these 'solutions' however were mainly implemented in the respective modules (such as for node language, the language storage, filtering, administration was all in node module, only a little piece of UI was in locale module). In Drupal 8, our approach is that we don't attempt to solve language problems from the outside, so node, path, etc. related code now resides in the respective modules. On last week's sprint, we got the remaining two pieces, localized date formats and field language moves to their respective modules, leaving locale module purely for interface translation (given the base language functionality is in Language module). This should let us focus this module only on this functionaliy and clean it up as such. The two issues related were http://drupal.org/node/1630264 and http://drupal.org/node/1635134.

Brand new interface translation translation interface

Drupal 7's built-in interface translation interface is a flash of the past, where you likely only had your Drupal sites in a couple languages that you all knew and you wanted to translate things all at once to multiple languages. That is not really the major use case anymore. So we took a fresh look at the translation interface and rebuilt it from the ground up. Now it focuses the user on one language and lets users translate multiple strings at once. It also builds-in singular/plural string pairing into this user interface, that was not even possible on Drupal 7. User reaction so far has been overwhelmingly positive. We've been working on this at http://drupal.org/node/1452188

New, highly flexible node type language defaults setup

Drupal 7 has language support for node types hidden under 'Publishing options'. There are three options for node language support. There is 'no language support' in which case, nodes are saved as 'Language neutral' and a language selector never shows. There is 'language support enabled' in which case the node will show a language selector and default it to the site language and this can be extended with 'translation support', which lets users translate, if the node had a specified language. This is extended a great deal in Internationalization module, but the setup is not very clear.

In Drupal 8, you now find language setup under a dedicated vertical tab. You also now have a built-in language default configuration user interface that is a lot more flexible. Now you can separately set the default language and whether to show/hdie the language selector, so you can set up configurations like default all your forum posts to a specific language or default node submission language to the user's preferred language instead of the site's. We are working to generalize this to the entity level, so you could have the same configuration on taxonomy terms for example in the future. Our issue for this work was: http://drupal.org/node/258785

Drupal now tracks the status of your imported .po files

Drupal 8 already centralized .po file lookups to one directory, so you don't need to hunt around for your .po files under different modules and themes. Work is ongoing to build-in the functionality of Localization update module, so that your site can be fed with interface translations automatically from localize.drupal.org. Yet another step of that integration is now in core! Locale module will now track all files that it previously automatically imported, and will only import them again if the date of the files changed. This is purely backend functionality, and the rest of localization update integration will put a simple user interface on top of it. We've been working on this in http://drupal.org/node/1635084

Discussions and ongoing work

Configuration management with multilingual configuration

We've had some great and extensive discussions leading up to the sprint with key people on the requirements and the design for a multilingual configuration solution for Drupal 8. We have mapped out an option A (with variants AB and AA in discussion there), option B and option C. The various options used various storage models, call patterns, context handling, and so on. I've made this figure to summarize the situation and we had various meetings to discuss the options with Earl Miles, Greg Dunlap, Jose Reyero, Daniel Kudwien, etc. Going through what we need and how we can get that done, option B won out, with the caveat of working together with the context/plugins systems introduced in Drupal elsewhere. Read the whole proposal at http://drupal.org/node/1616594.

1. Translatable vs. multilingual: Our discussion evolved around supporting both translated (like site name) and multilingusl (like default front page path) values, which differ in conceptual approach to whether they are human language translation or just value storage keyed by language. All other options but option B only allowed for storing some values for different languages and would have caused developer experience issues.
2. Config storage: To potentially allow any configuration to be multilingual, we either need to change the structure of configuration on the fly, introduce language at all places (like field API did, and freaked developers out), or externalize language variants to the degree that they are still managed within CMI storage for staging and deployment but don't decrease the developer experience. Option B externalizes language storage to separate 'override' files for languages (and optionally allows for other realms to have different values).
3. Context: We agreed that for storage and retrieval we need some kind of context. If you just think about field settings, we need to (a) display an admin form for them in their original language, (b) display them as labels on the entity form in the interface langugae and (b) display them as content data in the content language on entity view. So we really need developers deal with some kind of language context either way, indepdent of which method we choose. Option B passes this context around without requiring each method and function to get a specific language argument.

We have also been busy helping to port some settings to CMI to serve as our use cases, such as site information (http://drupal.org/node/1496542) and contact form settings (http://drupal.org/node/1588422). Neither of those landed yet and needs help.

We've also attempted to convert the built-in language list to CMI for distribution, which sparked some debate as to whether this is even configuration at the first place. You opinion welcome on http://drupal.org/node/1632236.

Entity property translation

A cornerstone of the Drupal 8 Multilingual Initiative is to reduce the two entirely different options for content translation to one. Instead of node-copy based translations and in-entity field translations co-existing, we want to extend in-entity field translations to support all fields and properties and drop node-copy based translations (migrating your data to the new model). For this, the Entity API work convering property API and making all properties use similar storage to fields is key. Turns out this is a shared interest between us and the WSCCI initiative, and there was extensive discussion about the topic at the WSCCI Web Services Format sprint, so we got some great looking plans from there. People sprinted on making this happen, but it is far from ready and needs lots of hands. Let me know if you want to help!

There has already been considerable work to convert some accessors to support language, like the label() method on nodes and properties, see http://drupal.org/node/1616952, http://drupal.org/node/1616962 and http://drupal.org/node/1616972.

Entity sprint: http://www.flickr.com/photos/amazeelabs/7373299908/in/pool-1993022@N21

Gettext parser refactoring

Clemens Tolboom has been hard at work on this before, now joined by Peter Droogmans and Christian López, they worked out a great interface and class structure to make out Gettext .po file parsing and generation up to date. Our current code in Drupal 7 is as coupled as it can be, taking a file name and directly inserting to the database of locale module all at once. The refactoring work makes it possible to take input from other file formats or export to other file formats, and just generally makes the parser and generator up to date to current standards. There is still work to do on this front (at http://drupal.org/node/1189184), and we created this figure to show the structure of the parser to those interested:

Integrating project awareness to locale module from localization update

Erik Stielstra and Florian Weber were busy to integate more of localization update in core. While file status monitoring landed (see above), work is ongoing to make project identification and file download happen. Erik worked out lots of new test cases, that were not at all in the contributed module to prove everything works right, and he needs more hands to continue this work. See http://drupal.org/node/1627006 (project identification) and http://drupal.org/node/1632384 (project file imports).

Language-aware entity form controller

We've also been continuing working to introduce language awareness to entity forms, so we can dislay the same entity form in different languages for translation. See http://drupal.org/node/1499596 for more information.

Sprinting fun: http://www.flickr.com/photos/amazeelabs/7188067403/in/pool-1993022@N21 Gearing up for Munich

Drupalcon Munich is very close! In just two months from today, people will gather in Munich to celebrate Drupal's greatness, learn new things and keep improving the software and the community. We are not letting that opportunity go either. The Drupal 8 Multilingual Intiative is planning to have another big sprint with three days before and three days after Drupalcon session days (sprinting on both the training day and on the official Drupalcon sprint day of course). We ned people signed up sooner than later to figure out finances, location and tasks for you all. Come to our meetings every other Wednesday in the meantime to figure out what is hot and where can you help out effectively.

Bonus: tips for sprint organizers

This was a huge sprint, and nobody is born to be a good sprint organizer. I've had my bumps as well, and wanted to share some good tips to make your sprint successful, outside of the basic things like getting the right people attend.

• Don't just focus on building, focus on delivery. You can build lots of stuff at a sprint and then when everybody goes home and business priorities take hold, all that cool stuff will get stale and not get into practice. As the end of sprint gets close, focus more on reviewing what you had and get is committed instead of building a few more new shiny things.
• Continually engage people, ask them if they are working on something useful, if they are stuck, or need more help. Do a daily standup to have a round of motivation and help people figure out what is going on but also do one on one sync-ups throughout the day. A wasted week is a disaster but a wasted day is still pretty sad.
• Cross-check your people; teams form around topics, but there are times when certain members would love to go on to other topics where they can be more productive. Don't be stuck with a very long meeting or days of collaboration on an issue with multiple people if not all of them are engaged.
• Having space for multiple groups of people is great. Separate tables for separate topics help focus work on where things are. Regrouping throughout the day based on the work you do should be encouraged. When people sit with each other, work flows much easier.
• Get ice cream! Lots. People love ice cream! You cannot even imagine how much they love it. Also, coffee. Almost everybody you ask will say they prefer some kind of coffee, but those will be different kinds. Espresso, Americano, Latte, and so on and on. Don't underestimate the need for coffee.
• If possible, get sponsors for food and other related expenses. Whether you have someone cooking on the grill for you at night or a selection of menus for lunch at a good place on the street, you get worrying about picking a place, getting there, dividing the bill, and so on off the shoulders of your sprinters. This lets them continue being focused.
• If you have an open sprint with various people coming, not restricted to your experts only, be prepared with a plan for people coming all the time. Different people have different schedules that they can participate. Have tasks ready for people coming, get them pair up with existing sprinters and review their work (whether code or user interface).

Daily standup: http://www.flickr.com/photos/amazeelabs/7188059783/in/pool-1993022@N21

• Drupal 8 Initiatives

This week Nedjo Rogers and Rosemary Mann join me to talk about their Open Outreach distribution.

Here's what we talked about

• What is open outreach?
• Why did you decide to create Open Outreach? (was it a personal need? client need?)
• What does Open Outreach do?
• What are some of the features included?
  • http://openoutreach.org/features
  • Are these using the apps model or another?
• Who is contributing to Open Outreach?
• How can groups get Open Outreach?

Use Cases

• Who is Open Outreach aimed at?
• What are some current projects and focuses?

What’s in the future?

• CRM functionality
• Compare External (CiviCRM) and internal (custom)

Questions from Twitter:

• @tgroff: What are the benefits/differences between using http://drupal.org/project/debut_member vs. CiviCRM roles sync?
• @tgroff: Does http://drupal.org/project/debut_member provide membership renewal, expiration, or renewal reminders?
• @tgroff: Mostly like to know more about the "debut" modules. ex: can I just install http://drupal.org/project/debut_bio on any site and use it? etc

Submitted on Wed, 06/20/2012 - 00:00

Tweet

Last weekend the Drupal Developer Days 2012 were held in a warm, sunny, beautiful and relaxed Barcelona. This is the first of two or three posts about the sessions there.

The conference was targeted on developers and a lots of sprints were going on around and during this weekend. There were a lot of sessions so I can not cover all that happended during Dev Days.

Designing and building for the editor experience
 

This session of Andreas Sahle and Daniel Nolde was about optimizing the experience editors have on a Drupal site. We all know that Drupal can be a PITA when it comes to easy content editing.  They gave some good pointers to think about when building a Drupal site. We are developers and not editors, we think different! Start thinking in roles, like what would a content editor or a translator do most of the time and how can I make his life a little more easy.

But how do editors think and what do the do? Most of them will think of content as pages. “How will this page look”. We, as developers, tend to think more structured in blocks, menus, node content, views etc. We have to start thinking in editorial flows. See also the discussion on Drupal.org.

An important part to think about is the choice to generate a page automatically or let the editor create the page. Most of the pages in Drupal will be a mixture of both, like a node page with an automatic generated views block showing related content. The hard part is deciding in what will be automatic and what will be manual on a page.

Start testing early with user stories and personas. And the permission system alone is not enough. This is not a replacement for an interface strategy! The interface should be intuitive, fast and efficient, avoid confusion and clutter. You can find some good tips in the slides below. Some things are quite simple as putting the most important thing on top, thinking about closed or opened field sets etc. And a lot of improvements are coming to Drupal 8.
But there are a lot of things in Drupal that can be frightening for editors. There are just to many options. Taxonomy for categorizing, menu structure, switching options. Blocks come in standard, views, views content panes, node blocks, mini panels etc. The good thing about Drupal is that it is a framework and you can extend and change it to overcome a lot of the problems.
They also gave some good pointers for modules that can improve the editor experience.
For select boxes you have

• Multiselect
• Chosen
• Select or Other
• Multiple selects

For dates you have date_popup_authored, for maximum length validation there is maxlength.
Think about using client side validation. This improves the experience lot. And set the message on the form element where it happened and not on top of the page with inline messages or inline form errors.
Finding and selecting content can be a hell of a task in a Drupal site.  Auto complete for references fields and links is not always the best choice. But you can improve it with linkit. That module will not only show the title but also gives some meta data back.
Modules that improve the creation and ordering of node references are nodeconnect and references_dialog.
Daniel Nolde has created the content menu module, that improves the administration of menus a lot.
For a complete overview see the slides

Architecting Drupal Modules - Report from the front lines
 

Ronald Ashri talked about the choices that a Drupal developer has to make when building a new module. Because when you build a new module, the path is not always clear. Let's say you want to add a JavaScript library to your module. How should you do that? All the choices....

• drupal_add_js
• Managing JavaScript in Drupal 
• Hook_library
• Libraries API

Or how should I structure my files?

• Everything in the root
• .module, .info in root. Images in images, JavaScript in js, includes in inc etc.
• Some in root, some in directories based on history, module evolution, style changes etc

And how should I name my files?

• .module, .info, .inc, .install, .test
• Occasionally throw in .theme, .install.inc
• Sometimes .modulename.thing.inc, classname.inc
• Myconventionisbetter.inc

And what about storing data?

• My table via hook_schema + dbtng
• Entities are the way to go - always (maybe?)
• Fields, then a field group entity or node to add what you are missing.
• Is your data content or configuration.

There are of course some important things to take in mind when writing a new module like:

• Separations of concerns
• Decoupled
• Consistent

A good thing to do is to define a problem and design not in a Drupal specific way. Your module solves a problem. You should be able to describe that in generic terms in plain English, not in Drupalese. This keeps you from overlooking possible solutions you otherwise might had missed.
We need to address these issues in the community and develop some common patterns for building modules like:

• OO patterns
• Drupal patterns: separate UI from core module functionality
• Service patterns: where possible decouple interaction points within the same module

We need to work on the methodology of writing modules and collecting start collecting and documenting patterns, discuss conventions.
If you are interested in this talk to Ronald Ashri. He wants to get this going. You can find his slides here.
 

/* Talking about code */
 

Jakob Persson is not really a developer, but he is the chief knowledge officer of Node One. And his session was all about knowledge and sharing of knowledge. Now that Drupal shops are getting bigger and bigger, they have to think seriously about knowledge management. You don't want your developers to solve the same solution twice. Sharing and spreading knowledge in your organization is about the flow of knowledge. And that happens when people talk to each other. And remember, plain information is not knowledge. People have to pick up that information. People should talk to each other, that's the most important thing.

So what are the reasons people don't share there information? You can think of thins as:

• “Knowledge is power” - Mostly not a problem in open source companies.
• Missing awareness of knowledge – They don't know the have the knowledge
• Lack of time
• Missing reward system
• Missing knowledge management awareness

So what can you do to stimulate people in sharing knowledge? Remember that the people are the most important so look there first. It's 70% about people, it's 20% about processes and it's  10% about technology.
Some things you can initiate to let people talk to each other. are

• Community of practice
• Daily stand-up
• Pair programming
• Solution match-making -  match people that have a problem and people who know the solution.

For processes you can think of a script for training new staff.  And as technology you can think of

• Wiki - You need a moderator to keep content up to date
• Site document
• Browser notes, a cool script in Greasmonkey that shows annotations on a drupal.org module page from your colleagues. 

Let people know that sharing information is really useful. Try to find some use cases where it worked. Make people feel useful. Remember, we have mainly intellectual capital. It's all in the heads of the employees.

Thanks

It was a great conference with lots of interesting session and I want to thank all the organizers, sponsors, volunteers and attendees to make this an awesome Drupal event again.

ResearchDrupal, Drupal-Planet, Development, Dev Days

I've been working on RecommenderAPI, a general purpose recommendation engine for Drupal, for a few years now. In the meantime, I'm doing my graduate work in recommender system, social computing, and machine learning. In this article, I'd like to discuss what to look forward to in the next major release of RecommenderAPI.

A brief history

The RecommenderAPI module (release 6.x-2.x) started as a Google Summer of Code 2009 project. It was entirely written in PHP, and was simple, easy to use. However, soon it runs into performance bottleneck: PHP is not really designed to do complex computation that uses lots of CPU/RAM, and such computation shouldn't run on the Drupal production server anyways.

Therefore, in a GSoC 2011 project, I completely re-wrote RecommenderAPI in Java (release 7.x-4.x and D6 backport), and uses Apache Mahout as the underlying computational library. This solves the performance problem by using Java multi-threading, a large amount of memory with the "java -Xmx" option, and it can run on a separate machine or potentially on a Hadoop cluster. Also, I have started a cloud service (currently in alpha testing) for people to try out RecAPI for free.

However, the current version of RecommenderAPI still has some limitations, which will be addressed in the next major release.

Plans for the next major release of RecommenderAPI

Step 1: Code refactoring, re-branding the dependent module, and code-readability improvements.

RecommenderAPI uses the "async command" dependent module to facilitate integration with Drupal. The "async command" module has been falsely branded as an asynchronous work queue module. But in fact, its main purpose is to provide code building blocks to help developers build 3rd party applications (esp. big data analysis programs written in Java/Python/R/etc) on top of Drupal. I plan to rename it to be "Drupal Computing" module in order to make that point clear, as well as make other code refactoring changes (rename class names, etc) to improve code-readability.

Step 2: Search API integration, and/or other ways to push data to external destination.

The current version of RecommenderAPI integrates with Drupal through direct database access. Many people have shown concerns about this approach, and suggested that the right ways is for Drupal to "push" data to an external recommendation engine. "Search API" is the the only way I'm aware of that "pushes" data. The Feeds module only imports data; and the Services module allows other programs to "pull" data, but it doesn't push data to other programs. I wrote a hack that uploads data to the recommender cloud server through HTTP, but it's not done in a systematic way. In the next major release of RecAPI, I'll explore different ways (including SearchAPI) to push data for Drupal to other external programs.

Step 3: Content-boosted algorithm, and other state-of-the-art recommendation algorithms.

As of now, the RecAPI module and Apache Mahout only supports the Collaborative Filtering algorithms. In other words, it only takes into account user-item preference information and makes recommendations accordingly. It doesn't use any textual content features. This is a serious limitation for many content-rich or knowledge based Drupal sites. Also, both RecAPI and Apache Mahout don't support the "ensemble method", which is the winning algorithm in the Netflix Prize competition. In the next release of RecAPI, I'll implement both the content-boosted algorithm and the ensemble method, and contribute it either as a patch to Apache Mahout or directly to RecAPI itself.

More readings about different recommendation algorithms:

• README file in RecommenderAPI
• Adomavicius, G., & Tuzhilin, A. (2005). Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering, Vol.17(No.6), 734-749.
• Koren, Y., & Bell, R. M. (2009). Matrix factorization techniques for recommender systems. Computer, 42-49. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5197422
• Bell, R. M., & Koren, Y. (2007). The BellKor solution to the Netflix Prize A factorization model. KorBell Teams Report to Netflix.

Step 4: Hadoop cluster support, and other performance improvements.

RecAPI supports multi-threading, but it is still quite limiting: I have tested the RecAPI engine on a 6-CPU server, computing recommendations for a 10 million user-items rating datasets (72,000 users, 10,000 items), and a full run took 11 days 20 hours. The magnitude of recommendation computation process might require hundreds of CPUs in a Hadoop cluster to run simultaneously. Apache Mahout supports Hadoop, but it needs to be integrated with RecAPI.

There could be other performance improvements too: 1) trade recommendation quality for performance gain, 2) support incremental updates in addition to full runs (currently only the item-item algorithm supports incremental update). I'll consider all these performance issues in the next major release.

Step 5: Better integration with web analytics.

In order to make good recommendations, we need to track site users' preferential behaviors, such as their browsing history, ratings, or purchases. Currently, RecAPI reads these data directly from the Drupal database. But a more efficient, and perhaps a more powerful approach is to track these data using web analytics and then have RecAPI read analytics data. In the next major release, I'll explore how to integrate RecAPI with the Piwik module, or the Google Analytics module, or both.

Step 6: Full cloud service support.

The RecommenderAPI cloud service is running at http://recommenderapi.com in alpha testing phase. Together with the next release of RecAPI, I hope to provide a fully-supported cloud service as well. It'll have all the features mentioned above, and will work as the freemium model.

Step 7: Better integration with the Drupal Commerce module and Acquia Commons distribution.

RecAPI is designed to be a general purpose API framework to take care of complex recommendation computation, and it relies on helper modules to apply its computational power to real world scenarios. There are multiple ways to optimize RecAPI to work for eCommerce sites with Drupal Commerce, or social network sites with Acquia Commons. For example, we can combine product features, customers' browsing history, purchases, ratings, comments and demographics information together in order to make the best product recommendations. I'll spend some time to optimize these helper module too.

Timeline

I'm very excited about these planned items, but unfortunately I can't work on them right now. Most of my time nowadays are spent on my PhD dissertation, which is also exciting.

I expect to graduate in December, 2012, or January, 2013. Before graduation, I'll find some spare time to maintain the current version of RecAPI (fixing bugs/issues, etc.), and hopefully work on Step 1, Step 2, and part of Step 3.

After graduation, I'll work full time on all these items as a startup company. It'll take 6-9 months full time work to implement these features, so I'm quite optimistic to deliver them at the end of 2013. In the meantime, please stay with me and give it a little patience.

Vision

It is my personal commitment to build the state-of-the-art recommendation engine for Drupal, so that any Drupal website can use the power of recommendation engine out of the box.

A few last words about the benefits of using RecommenderAPI:

• It integrates with Drupal seamlessly.
• It's open source: you can use it for free, and customize it if needed.
• It's backed by top talents from in the field (such as my advisor Paul Resnick), so you can be sure the recommendation quality is good.
• We'll offer cloud service and premium support for those who don't want to directly deal with technical details.
• We contribute to Drupal and we want to grow with the Drupal community: support us and support Drupal!

Alternative title: Never create another virtualhost

Late last year, the board of the Drupal Association put six things on the agenda for 2012.

One of those was to coordinate Global Training Days to get more people to start learning Drupal and help address the world wide talent famine for skilled drupalistas.

The next one is happening on Friday 22 June. That's this week.

The dates for follow up Global Training Days are 14 September and 14 December.

See drupal.org/learn-drupal for all the details

 

But how did we get here?

In January, Dries spoke briefly about the Drupal Association at Drupal Downunder. Here's a snippet

And in February - the global drupal community stepped up by running events in Africa, Europe, Japan and the United States. On Friday this week, Australia, the Ukraine and Columbia joins in too.

Can you help spread the word?  Get out on twitter. The hashtag is #learndrupal - let people know it's happening, and gear up for the next one in September.

If you want to attend, register online direct with the training partner running the day.  See the full list to see if there's a Drupal training day somewhere near you.

drupal.org/learn-drupal

• drupalplanet