AustroFeedr - Real-time Open Data Aggregation, Processing and Publishing in Austria
AustroFeedr creates a universal & open source prototype for aggregation, processing and data publishing in real-time based on the PubSubHubbub-protocol. By implementing the „AustroFeedr Hochwasser“ (flooding) real world use case in cooperation with the Austrian government, it showcases the technology as a monitoring system of river water levels for flood protection in Austria.
Check out the features of AustroFeedr at our live demo platform: http://demo.austrofeedr.at
Real-time updates play an important role in nowadays web. From catastrophe warning messages over civil services to Twitter and Facebook and location-based services gowalla – they all rely on instant update notifications. Standard „pull“ RSS-Feeds require the client to poll for updates while the open protocol standard PubSubHubbub allows for a much more efficient „push“ strategy in order to send update notifications to subscribers. Big players like Google already incorporate this standard and experts like Tim O'Reilly call it „The Next Big Thing“.
Overview of river water levels at survey stations on a map in Austria with a pop-up showing detail information with a chart:
Flood level warning message with status, line chart and map information:
Flood level warning message delivery to the mobile:
AustroFeedr flow chart: Aggregation, Processing & Publishing of Open Data:
Related News from external RSS-Feed:
Open Data Export as table and CSV-download:
- AustroFeedr allows to aggregate open data from external publishers, process it internally and publish it in real-time using the PubSubHubbub-protocol.
- „AustroFeedr Hochwasser“ (flooding) is a real world use case implemented in cooperation with the Austrian government in order to visualize water levels for flood protection in Austria.
- Updates are published as real-time-enabled RSS-Feeds. Additionally the web site provides visualization techniques as maps and diagrams.
- Aggregated data may be categorized and filtered automatically. The processed data is therefore published within different channels to which user may subscribe.
- Regional classification of content allows AustroFeedr to display relevant news per region and district in Austria.
- Users may receive updates on their Android smart phone in real-time. iPhone support can be added transparently by making use of the PubSubHubbub-protocol.
- The website http://www.austrofeedr.at provides documentation for the project.
In cooperation with the Austrian government, we created a use case that implements AustroFeedr functionality in order to provide a flood warning system. As water levels rise quickly, affected civilians want to get informed instantly. We aggregate water levels and related news, process them internally and publish the data as warning notifications to the mobile and interactive visualizations on the web site.
- Warning message
- Geographic context
- Water level
- Regional news and updates
AustroFeedr was iteratively developed from September 2010 till September 2011. We decided to build on Drupal 7 which was released at the beginning of 2011 in order to make use of new features like Entities and Field API integration in Drupal core.
Building AustroFeedr as a distribution with a make file and features allowed to streamline the development process with a local dev environment and a staging server kept in sync by controlling the features with the versioning system Git.
AustroFeedr base functionality have been separated from use case specific functionality in order to provide both a generic base installation of AustroFeedr and on top of it develop the specific Use Case extending on it.
The data structure for AustroFeedr and the flood prevention use case where implemented using custom content types and the Field API which replaces the CCK module since Drupal 7. We used the DataStore module in order to store timestamp/value pairs in a performant way.
Data – Content Types
The AustroFeedr Features set defines a Subscription Feed content type which is used with a Feed importer for aggregating content from RSS feeds. The Update content type stores according messages which can then be published in real-time using PubSubHubbub.
On top, the AustroFeedr Hochwasser Features include content types for storing water level survey stations which contain reference data provided by Austrian government and store datestamp/value pairs for the water levels aggregated periodically. Flood warning messages are generated when a river's current water level exceeds a certain limit.
Flood level warning message with status, line chart and map information:
Aggregation – Feeds & Web Services
AustroFeedr aggregates information from different sources both in real-time and by classic pull requests. Real-time aggregation is provided by the Feeds module which allows to subscribe for PubSubHubbub-enabled feeds.
Processing – Rules & Rules Autotag
Water levels imported from the XML Web Service will be processed directly by associated Rules components. They update the status for the water level survey stations and generate warning messages if appropriate.
We mash-up news with regions by automatic categorization of imported news items using the Rules Autotag module. In order to filter for relevant data, it compares the full-text of news articles to taxonomy vocabularies of Austrian regions/districts and flood prevention terms.
Publishing – PubSubHubbub & Visualization
AustroFeedr publishes aggregated and internally processed data in real-time using the PubSubHubbub-protocol. Supplementary visualization techniques include maps and diagrams.
Every water level survey station lists a histogram of water level changes over time using the DataStore visualization component.
We successfully created a feature-rich prototype for aggregation, processing and publishing of open data in real-time using Drupal with a real-world use case for monitoring water levels to create a flood prevention system in Austria.
During the project we created features to cover data structures and aggregate content using Feeds in real-time & Web Service Client. Further they include processing components using Rules and content classification with Rules Autotag. Created content can be published in real-time using PubSubHubbub and displayed visually on maps and diagrams. All those functionalities together represent the feature-richness of Drupal with its multitude of contributed modules.
Open data in Austria is getting big recently, as initiatives like Open3, Open Government Data Wien and Linz Open Data indicate. AustroFeedr was present at various related conferences like Gov2.0Camp 2010 Vienna, BarCamp & Mobilecamp Vienna 2010 and Open Government Data Conference 2011. Planned future activities in that area include CreateCampWien a11y opendata and Gov2.0Camp Vienna 2011. Besides engaging with open data communities, AustroFeedr also connected with the Drupal Community as presentations at Drupal Austria Meetup, Alpine DrupalCamp 2011 in Brixen and within Drupal Tour Central America 2011 show.
We very much enjoyed collaborating with the Drupal community. Most of AustroFeedr's functionality is based on the great variety of Drupal modules available, so we didn't have to reinvent the wheel but rather assembled the features using advanced modules like Field API, Entity API, Feeds, Rules, Views and OpenLayers. A main focus of the project was therefore testing alpha and beta releases of modules. This made us contribute at about 150 issues from bug reports over feature requests to actual code patches.
Furthermore than collaborating on existing modules we are proud to contribute new modules to the community as a part of the AustroFeedr project. See Rules Autotag and the Pusher and PubSubHubbub Hub D7 versions for Drupal 7.
Especially exciting and motivating was focusing a main part of the project on creating the real-world use case for AustroFeedr. The use case implements a flood warning information system which incorporates all functionality from aggregating data over processing it and finally publishing the data in real-time and different visualization techniques. The successful cooperation with involved Austrian government entities as Lebensministerium and LFRZ allowed to access data for river water levels in Austria using Web Services. By open sourcing the results of AustroFeedr we are looking forward to applying the technology for real-world public use cases.
The high complexity of using 33 Drupal 7 modules and 20 AustroFeedr-specific feature configurations both meant a great opportunity for the project to showcase state-of-the art Drupal functionality but on the other hand it was challenging as well due to the early stage of some modules in spring-time 2011.
Most encountered problems could be solved within the team of epiqo and the Drupal community. The testing of complex scenarios as the real-time publishing in combination with web service aggregation and internal processing was quite challenging though and took longer than planned. We therefore skipped performance tests for the AustroFeedr prototype which we find crucial to actually take the system productive.
Code published as open source on drupal.org
- Hochwasser Demo - live use case demonstration web site
- AustroFeedr - base installation profile
- AustroFeedr Features - base features
- AustroFeedr Hochwasser - use case installation profile
- AustroFeedr Hochwasser Features - use case features
- AustroFeedr Water - use case theme
AustroFeedr Poster at Open Government Data Conference 2011:
AustroFeedr Presentation at Drupal Tour Central America 2011: