OSSEval, PROSE Open Source Evaluation Methodology and Tool


by Davide Galletti 2678 views

OSSEval, PROSE Open Source Evaluation Methodology and Tool

Software procurement – the process of evaluating, selecting and obtaining a software – plays an important role in ICT.
Given that PROSE’s main goal is to promote the use of open source software in within the ICT ecosystem, we developed a methodology and implemented the tools to run an educated open source software procurement process.
The present blog post first summarizes why a similar methodology is needed, and then it briefly describes how that works.

Open Source Software Procurement: why is that difficult?

Companies and organisations run software procurement by matching products functional and not functional characteristics against their needs.
Marketing material as well as direct access to sales reps it is often part of the decisional process.
Unfortunately within the open source world there is a lack of marketing material, with very notable exceptions (i.e. the most famous open source ‘brands’).
Since features, security and performance testing are time-consuming, before analysing them in depth it worth to identify a method for finding eligible candidates, limiting hands-on activities to a very limited set of projects.
Actually to qualify and select open source software a number of open source specific metrics need to be measured, ranging from the sustainability of its community to the easiness to contribute to the code.
We have been looking at a number of methodologies aimed at providing a solution for above mentioned issues, and we believe SOS Open Source – an automated methodology that gathers and analyzes data about open source projects, providing a synthetic representation of all selected candidates and also a graphical tool to compare them – was valid starting point to make educated choices.
Actually SOS Open Source analysis returns a set of open source candidates that are robust (stable, mature and backed by a viable community), supported (either by a community or vendors) and grant evolvability (readable and maintainable code).
We eventually come up with the decision to improve both SOS Open Source methodology and tools, eventually integrating it within the portal under the OSSEval name.
OSSEval adds value to the platform, it provides hosted projects with a tool to assess existing open source artifacts as well as asessing their own deliverables.
OSSEval can be used by the project promoters to address potential customers’ concerns (comparing their products with similar open source products); can be useful for IT Decision Makers performing procurement activities, and finally for reviewers who can use it to understand the intrinsic value of deliverables and areas of improvement.
So unless you are going for the obvious names or your roots are deep in the open source world – either if you are a user interested in using open source (borrow or buy, see D3.3 for references) or a maker willing to build software using it – you better take your time to ponder before selecting, and to try before buying or implementing.
All your typical needs, that will include the desired level of support, stack certifications, etc. must be carefully assessed and taken into account at evaluation time.
As FLOSS is used inside FP7/H2020 projects, both developed and adopted, these needs are relevant for both projects developing software, as well as adopting software.
The capability of exploring the relationship between procurement and FLOSS hosted at, can contribute directly to the long term success of the hosted software, as well as the platform itself.
Therefore SOS Open Source methodology, now integrated with as OSSEval, is extremely relevant to PROSE goals, and can be a unique value proposition for the platform going forward.


OSSEval methodology defines criteria to assign scores to each element, and the final result aggregates those scores up the tree of the model that defines the following three major frames:

  1. Project Sustainability (what a project needs to stay diverse and productive);
  2. Project Industrial Strength (what makes a project ready for prime time in a enterprise env);
  3. Project Strategy (what makes a project a valid strategic decision for the adopter).

For example, among the 7 different metrics covered by the Project Sustainability major frame, you find:

Code maturity [ less than 1 year, 1 to 3 years, more than 3 years ]
How to compute: browsing forges/meta-forges (please note that sometimes projects are not released in open source from the very first day; moreover source code could be moved from a forge to another forge);
How to improve projects’ marks: no shortcut, time is definitely not compressible.

Case study availability [ unknown, case studies available only on the website, case studies available on the net ]
How to compute: using search engines on specific sites and on the net;
How to improve projects’ marks
: publish case studies, possibly in different languages, invite end-users and customers to write their own case studies and promote those; make sure other end-users or customers are going to publish their own experiences.

The assessor evaluating an open source components won’t be tempted by bypassing any step because the whole assessment takes less than an hour.
Besides automatic support provided by the tools, the assessor can leave comments.
If the assessor chooses a value for a metric that differs from the one suggested by the system she can track the reason of such decision.
An OSSEval assessor can evaluate a project in half an hour on average.
Draw up a short-list of 6-8 candidates, evaluate and compare them takes about 2 days.
All functional analysis and testing maybe performed then on a very limited set of projects, saving time and resources.
OSSEval simplifies all tedious tasks and the assessor does not need to be an analyst nor a geek, all questions are supported by automatic tools returning values for metrics or search results useful to answer all metrics’ questions.

OSSEval in pills

The first step is to identify a shortlist of potentially suitable products.
To start your search you may want to start by checking the availability of software programs related to the category to which the program belongs to. In this sense ‘trove‘, the SourceForge taxonomy made available with a Creative Commons license, it is probably the most comprehensive and one of the most exhaustive.
Once we are clear about which is or which are the most appropriate categories we can start looking for open source projects falling in those baskets, either using forges, meta-forges and public directories like Wikipedia and dmoz.
The second step is about choosing the purpose of use of the program you are looking for.
The purpose of use of a program may in fact determine a filter more or less selective, something OSSEval supports allowing assessors to set different ‘weights’ for different metrics or – even more simply – picking up a specific ‘profile’ where a profile is a set of predefined weights tailored for specific needs.
For example a semi-mature application can be adequate for a trial, but it won’t probably fit if you are building mission-critical applications. The intended use also can be enriched with:

  •  - the type of end-users (internal, partners, customers, individuals or businesses);
  •  - the level of outsourcing (internally driven development, using external resources in man-power or outsourcing);
  •  - intellectual property protection requirements for trademarks, copyright and patents.

We can think of at least two different profiles, namely:

  1. Borrowers;
  2. Lenders.

In the first group we have people willing to reuse code provided by third parties.
They are not planning to enhance or extend what is provided, and they have basic requirements for Intellectual Property (IP) management.
In the second group we have people who share what they do, they need to mind their steps in terms of IP rights, and that could be easier or simpler depending on their development model (insource vs outsource).
For both borrowers and lenders it’s important to establish if the software is going to be used for prototyping or for mission-critical. We can have two profiles with two sub-profiles each, representing low/high importance on the level of industrialization achieved by the program.

Comments (0)

Comments are closed.

PROSE will contribute to the adoption of open source software on ICT projects, by increasing the lifetime of the software developed inside European projects and thus maximizing impacts. This will be achieved through the creation of a coordination platform for hosting software projects, as well as promoting dissemination and training events on Open Source topics.