Scality open source program

First Drops released from Scality's Bounty Program

San Francisco, CA - January, 30th, 2011 – A few months ago, at SNIA's SDC in Santa Clara, we launched our open-source initiative under the name SCOP (Scality Open Source Program). Within this program, we released an open-source library called LibDroplet, helping developers underpin their application's storage needs with StaaS providers.

 At the same time, we also started a $100,000 bounty program allocated to multiple applications and called out to developers to work on select projects. This program was designed as a win-win-win situation in which:

• The community benefits from new cloud storage-enabled applications (released as open-source),
• Scality's customers benefit from new applications to interface with their RING StaaS platform,
• And developers are rewarded for their effort.

Today, we're proud to announce the immediate and general availability of two of those applications developed using LibDroplet: DropFuse, and DplSh, described later. We're also very happy to announce the developers selected for the next batch of applications.

LibDroplet is a very streamlined library that includes all that is needed to handle online storage: simple and uniform storage mechanisms, usage statistics (number of objects stored and operations made, in and out traffic usage) as well as billing information (configurable per storage provider). This library aims to be a complete abstraction layer between applications and online storage providers of any sorts. It relieves developers from having to deal with different protocols and let them focus only on storing information, knowing it will be accessible in a uniform way across applications and providers.

Originally under the GPL, LibDroplet is now re-released under a BSD-compatible license in the light of Scality's involvement in SNIA's CDMI initiative. To the community, LidDroplet is still completely open-source and available for everyone to use, reuse and modify at https://github.com/scality/Droplet

 

First Drops released

DplSh is the first Drop to be available to the general public. It allows anyone holding an account on a compatible cloud storage provider to manage their objects through a Command Line Interface with familiar commands like managing buckets and objects (with a nice server-side copy – inside a bucket or between buckets – mechanism). It also leverages LibDroplet’s Virtual Directory layer (VDir), allowing the creation and manipulation of directories, something not usually inherent to cloud storage.

We're also proud to announce the immediate availability of DropFuse, a high-level file system connector for Online Storage accounts. Using DropFuse, one can locally and transparently mount their buckets as a regular drive / mountpoint . It also features an asynchronous write cache and in-line compression to enhance performance and minimize bandwidth and operations cost.

« DplSh and DropFuse are great examples of how LibDroplet can accelerate the development of cloud applications. Security and inter-operability are the key to widespread adoption of cloud storage, and I’m thrilled about the response on LibDroplet from the community so far.», commented Giorgio Regni, Scality's CTO.

On top of that, there were 10 other Drops specified on SCOP's website (http://scop.scality.com) at the time of launch, and amongst the many submissions, we have selected what we consider the best applicants for 5 of those applications.

• Cloud Migration – David Pineau
• KVM Virtualization  - Simão Reis
• Gnome GVFS and Nautilus Extension - Rui Tiago Cação Matos
• WordPress & Gallery3 - Gary Mort

Even though the original bounty program is officially over, since we had such a good feedback from the community about it, we decided to leave it open for the time being, so if you feel like coding for the community, feel free to join the band by submitting your applications to one of the drops left to be done:

• Zope/Plone Interface
• Alfresco CMS
• FastCGI CDN
• Datanucleus ORM Platform Datastore
• Linux Virtual Tape Library

We are looking forward to seeing those Drops being released and available. Please follow us on twitter (@scality) for updates, on GitHub for more detailed information, and "drop" us an email at drops@scality.com if you want more information.

The Scality SCOP Team

Datanucleus ORM platform datastore - [sd_datanucleus]

Apply for this bounty

Overview

The goal is to extend the S3 support available in datanucleus (http://www.datanucleus.org). Datanucleus is an ORM platform following the JDO specification : it is a set of tools that transparently generates the code necessary to store and retrieve java objects in various datastores (RDBMS, LDAP, S3). Once persisted, objects can be retrieved using a query language - JDOQL - that heavily borrows from sql but that is better suited in an object oriented. A key advantage of SQL is that it is entirely evaluated in the datastore and that it is hence very efficient.

At the present time, datanucleus includes a limited support for S3 (http://www.datanucleus.org/products/accessplatform/amazons3/support.html). However its accompagnying JDOQL implementation is rather inefficient since all candidates objects are retrieved from datastore, and then evaluated in-memory, obviously limiting the practicality of S3 for java object persistency. This is different from the JDOQL implementations for RDBMS where the JDOQL is completed translated in SQL.

The goal of this extend the S3 support available in datanucleus so that JDOQL queries does not require to load the full set of candidate objects in memory using Scality droplet library.

Scenarios

S3 supports a notion of hierarchy that can be used to build a kind of index (http://github.com/scality/Droplet). This feature can be used to build indexes. Query should run against these indexes, hence reducing the amount of objects retrieved in memory. JDO uses metadata data to specify the relationship between objects persisted in the datastore. Primary and foreign key should be indexed automatically. In addition, the index tag should be supported.

Non-goals

It is not the purpose of this project to enforce referential integrity within the S3 datastore at this time. Instead this project focuses on the scalability of the S3 datastore.

User interface

This extension must be tightly integrated within the datanucleus platform.

---

Codename: [sd_datanucleus]

Bounty: $5,000

Apply for this bounty

KVM virtualization storage engine - [sd_linuxkvm]

This Bounty has already been assigned to a developer: Simão Reis.

Overview

The goal is to add a cloud storage option for KVM, a full virtualization solution for Linux on x86 hardware.

It will provide highly available block level storage volumes that can be attached to KVM virtual machines. The solution should not require any central node, for example, no central meta-data server and provide a completely stateless operation model. This will allow VM to grow on demand and keep fresh data locally, in a cache, and upload old pages of data transparently to the cloud using Scality Droplet library.

Scenarios

A virtual machine is setup as usual with virtual network interfaces, cpu and memory. A number of virtual disks can be configured, a new type of virtual disk is available, cloud based storage. A mix of regular local storage and cloud based storage is possible. For example, a vm with a root partition on local disks and a /mnt partition on the cloud with a 20GB maximum local cache.

User interface

This storage engine should fully integrate into kvm usual configuration files as new virtual disk option.

Other features

This storage engine must make use of key Scality droplet features:

• Encryption: data is encrypted before being sent to the cloud
• Local caching: a local folder is used as a cache, either read cache only or write back cache for the highest performance.

Non Goals

There's no need to re implement a LRU type cache as Scality droplet already has one.

---

Codename: [sd_linuxkvm]

Bounty: $10,000

Assigned Developer: Simão Reis

Gallery 3 Plugin - [sd_gallery]

This Bounty has already been assigned to a developer: Gary Mort.

Overview

Gallery is a popular photo album organizer. Gallery uses a combination of a RDBMS and traditional file-system storage. The Gallery model seems well adapted to use object based storage in place of traditional file system storage.

Scenarios

The key feature being requested is the full replacement of the underlying filesystem based storage of content/objects with object storage using the REST interface of the Scality Droplet library.

Presumably a php based plugin or code modification will be used to introduce this functionality.

Non Goals

The objective is not to replace Gallery or its RDBMS, but to enable cloud storage and serving of photos via Gallery.

Slowing down Gallery is obviously not an objective, this integration should allow scaling up to very large collections of data without hurting performance.

User Interface

Gallery user interface is not expected to need specific modification except to enable the following controls:

• An interface to configure and test the cloud storage provider credentials and hostname.
• The ability to generate, publish, and respond to storage pricing information

Extras

Support for Gallery 2.

The php bindings required to do this development could prove very interesting to many other projects. Well designed php wrappers for the Scality Droplet library could be integrated into the Scality opensource project for the use of others.

---

Codename: [sd_gallery]

Bounty: $2,000

Assigned Developer: Gary Mort

GitHub Project URL: https://github.com/garyamort

Linux Virtual Tape Library - [sd_linuxvtl]

Apply for this bounty

Overview

Linux Virtual Tape Library is an open source implementation of a VTL. Each virtual tape is composed of 3 files, a data file, an index file and a meta file. It seems reasonable, at least for reasonably sized tapes, to benefit from cloud storage flexibility and pay as you need cost model.

Scenarios (use cases)

The feature being requested is the implementation of object based cloud storage using Scality Droplet client library.

Non goals

The goal is not to show the ability to implement a large number of virtual tape drives.

User interface

Configuration files should allow:

• Configuring and test the cloud storage provider credentials and hostname.
• The ability to generate, publish, and respond to storage pricing information

Extra

Demonstrate the implementation working with various backup packages.

---

Codename: [sd_linuxvtl]

Bounty: $3,000

Apply for this bounty

Alfresco CMS plugin - [sd_alfresco]

Apply for this bounty

Overview

Alfresco is a CMS (Content Management System) that uses a combination of a RDBMS and traditional file-system storage. The storage model seems well adapted to use object based storage in place of traditional file system storage.

Scenarios (use cases)

The key feature being requested is the full replacement of the underlying filesystem based storage of content/objects with object storage using the Scality Droplet client library.

Non goals

The objective is not to replace Alfresco or its database but to enable cloud storage of objects via Alfresco.

Slowing down Alfresco is obviously not an objective, this integration should allow scaling up to very large collections of data without hurting performance.

User interface

Alfresco user interface does not need specific modification, although the following controls should be made possible:

• An interface to configure and test the cloud storage provider credentials and hostname.
• The ability to generate, publish, and respond to storage pricing information

Extra

Ideally Alfresco could use the persistent storage of the RDBMS and the object store in a stateless fashion allowing multiple instances of Alfresco to be launched as required connected to the same database and storage. Additional credit will be given to a solution capable of providing this type of fully distributed stateless architecture.

---

Codename: [sd_alfresco]

Bounty: $2,500

Apply for this bounty

Wordpress Plugin – [sd_wordpress]

This Bounty has already been assigned to a developer: Gary Mort.

Overview

Wordpress is a very popular tool for blogging and website creation. Some Wordpress installations are shared with numerous users, and data volumes can become significant. The content managed by Wordpress is typically stored locally on the server running the server process, or possibly on a SAN, NAS or other network attached storage. The native HTTP nature of web interactions makes object based storage an obvious choice for storing objects to be served by web server.

Scenarios

The php based plugin will provide all backend storage for pages, images and other media delivered by the Wordpress instance via the Scality Droplet library connected to the user's preferred cloud storage provider.

A Wordpress user can deploy the cloud plugin individually without requiring all users of the Wordpress instance to use the plugin.

A script will permit migration of existing content from the local directory to the cloud storage system.

Non Goals

• Changes to source markup should not be necessary and formatting of content should not be adversely affected
• Performance should not suffer, caching on local disks of frequently accessed posts might be required.

User interface

A user form for the configuration and test of the cloud provider host address and user credentials will be required

A well formatted log or status page showing the costs incurred for the use of the storage.

Extra

The submitter is free to suggest additional functionality and features deemed useful.

The php bindings required to do this development could prove very interesting to many other projects. Well designed php wrappers for the Scality Droplet library could be integrated into the Scality opensource project for the use of others.

---

Codename: [sd_wordpress]

Bounty: $2,000

Assigned Developer: Gary Mort

GitHub Project URL: https://github.com/garyamort

Fuse file system high-level plugin - [sd_fuse]

This Bounty has already been assigned to a developer: Guillaume Leconte.
It has since then been released: https://github.com/pozdnychev/dropletfs-fuse

Overview

Filesystem in Userspace (FUSE) is a Linux kernel module that lets non-privileged users create their own file systems without editing kernel code (example sshfs, ftpfs, even gmailfs...).

The goal is to write a high-level fuse file system "dropletfs-fuse" for cloud storage through Scality Droplet library.

Scenarios (use cases)

The key feature being requested is a high-level FUSE plugin which mounts to a S3 bucket and uses the Vdir/Vfile interface of the Scality Droplet library to manipulate paths.

File system attributes (uid, gid, mode, etc) shall be stored by using the metadata interface of Scality Droplet.

Manage all special files (e.g. symlinks) but it's not required to manage hard links as those create significant difficulties for properly handle reference counting.

Non goals

The goal is not to write a fuse low-level interface neither to rewrite your own path translation mechanisms. It is important to leverage the virtual directory extension of Scality Droplet library (and not only displaying "raw" files contained in the bucket).

Hard links support isn't required.

---

Codename: [sd_fuse]

Bounty: $2,500

Assigned Developer: Guillaume Leconte

GitHub Project URL: https://github.com/pozdnychev/dropletfs-fuse

Gnome GVFS and Nautilus Plugin - [sd_gvfs]

This Bounty has already been assigned to a developer: Rui Tiago Cação Matos.

Overview

GVFS is a filesystem abstraction library used by Nautilus file viewer, allowing applications plugable transparent access to a variety of "real" filesystems, from WebDAV, to the local filesystem, or various URIs as well (http, ftp, ssh, etc).

The goal is to write a s3:// URI gvfs module used to access an S3 compatible cloud storage.

The goal is also to write a Nautilus extension to perform S3 service operations (bucket creation, price display, etc).

Scenarios

The key feature being requested is a gvfs module which defines the cloud:// URI plugin by using the Vdir/Vfile interface of the Scality Droplet library.

When the user types an cloud:// URI in Nautilus, the latter shall ask for an account specification and propose to store it in a profile (like the ssh:// URI). Once authenticated, it browses buckets, files and directories of the given account.

The user shall be able to create buckets directly through Nautilus and get statistics about the subdirectories displayed, including monthly pricing information. Files metadata should be taken into account and user should be able to get and set them.

Non goals

The objective is not to replace gvfs or nautilus but to enable cloud storage via gvfs and Nautilus user interface. This way any gvfs-enabled application will be able to "mount" or access files sorted in the cloud transparently.

Extra

It is important to leverage the virtual directory extension of Scality Droplet library (and not only displaying "raw" files contained in the bucket). Then, it shall be possible to browse S3 buckets and all the sub-directories transparently.

---

Codename: [sd_gvfs]

Bounty: $2,000

Assigned Developer: Rui Tiago Cação Matos

FastCGI CDN Specification [sd_fastcgi]

Apply for this bounty

Overview

Web servers are ubiquitous with millions around the world. The content served up by web servers is typically stored locally on the server running the server process, or possibly on a SAN, NAS or other network attached storage. The native HTTP nature of web interactions makes object based storage an obvious choice for storing objects to be served by web server. A FastCGI architecture is well adapted to replacing local filesystem storage with object based cloud storage.

Scenarios

A FastCGI directly delivers objects stored in a local or remote cloud storage via the Scality Droplet library. The Web server instance must be able to perceive and manage data in the same way it would with traditional file storage.

ACLs and other attributes obtained and used from traditional filesystems should be faithfully reproduced. Preference will be given to generality and compatibility for the FastCGI.

Apache, Lighttpd and Nginx are considered core FastCGI compatible web servers on which compatibility should be verified.

Non Goals

• Changes to source markup should not be necessary
• MIME types based on filename extensions should be maintained
• Performance should not suffer, caching of what would traditionally be directory listings may be required by the FastCGI process.

User interface

Guidelines for appropriately storing data and naming constraints (if they exist) should be clearly identified.

Extra

The submitter is free to suggest additional functionality and features useful for web publications, service and delivery.

---

Codename: [sd_fastcgi]

Bounty: $2500

Apply for this bounty

Cloud Migration Tool Specification - [sd_cloudmig]

This Bounty has already been assigned to a developer: David Pineau.

Overview

Nobody wants to feel trapped by a storage provider, especially when cloud storage is a really fast growing market segment and competitors are bound to emerge left and right. Users want to get the best price and performance and that may very well mean switching service providers at some point in time.

A cloud migration tool will make it as easy as pointing to the old service provider, the new one, entering all necessary credentials and start copying data around.

Since we are working over the internet and transferring very large amount of data, verifying data integrity and making sure the whole process can restart where it left off in case of failure is mandatory.

Scenarios

Moving data from cloud A to cloud B

• data is read from cloud A
• written on cloud B
• State information is stored to be able to restart where the transfer left off in case of any issue.

When all data has been copied, another command should be started by the end user to delete all data from cloud A. This has to be triggered explicitly by the user, this should not happen automatically.

Ideally state information is stored in the cloud, maybe in a dedicated bucket so that local machine state is not required to resume/monitor a transfer.

Non goals

This isn't a two way remote sync. Its only goal is to copy data to another cloud and delete data from the source when everything has been successfully copied.

User interface

This is a command line based utility, it should display information about how much data is to be copied/migrated as well as the copy speed and ETA. The command scp from the openssh package is a good description of what the user interface for this tool should be.

Extra

Volume of data

Please note that the volume of data to be copied can take enormous proportions. We are talking terabytes of data and a single transfer could span over multiple days or weeks.

---

Codename: [sd_cloudmig]

Bounty: $7,500

Assigned Developer: David Pineau