Next 1st of July we will attend & participate in the 2nd workshop organized by the GraphLab. The aim of the workshop is to bring together people from the academia and scientist from the industry that have a special focus of large scale machine learning on sparse graphs.

The preliminary agenda can be checked here: http://graphlab.org/graphlab-workshop-2013/preliminary-agenda/

Join us there!

We would like today to share a very interesting article published by Nicolas Dugué and Anthony Perez from the University of Orleans about the detection of social capitalists on Twitter.

Social capitalists are those users that try to gain visibility by following users regardless of their content. Social capitalists are not healthy for social networks as they help spammers to gain visibility and may mislead influence detection.

In the article, they show that social capitalists can be detected using similarity measures, and that there is no need to analyze the tweets of the users, but rather the graph topology.

Another aim of the research was to focus on efficient & high-level techniques to store and handle very large graphs. After unsuccessfully evaluating SQL and other NOSQL technologies, such as Cassandra, they moved to graph databases which are better suited to quickly answer questions like retrieving the neighborhood of the nodes, which is essential in the computation of their algorithms. They chose DEX because in their own words “meets their requirements of efficiency and storage of large graphs” and “appeared as viable for several reasons: high-performance and graph oriented, a high-level API, and well-documented”. Nicolas Dugué and Anthony Perez research uses the Twitter graph, a spam graph and a list of 100.000 potential social capitalists: “using DEX we were able to store a graph containing about 15M vertices and 1B arcs”.

Some of the techniques used by social capitalists are “follow me and I follow you” or “I follow you, follow me”, making that the most of the users they follow should follow them back (overlap). On the other side, spammers wish to accumulate followers and then spread spam links. A previous paper about link-farming on Twitter and focus on spammers by Gosh et al introduced social capitalists as the users who most respond to request by spammers. Nicolas Dugué and Anthony Perez, use the previous results to contrast theirs using the proposed new faster detection techniques, obtaining an even bigger list of social capitalists.

To learn more about the detection of social capitalists, we highly engage you to read their article here: http://link.springer.com/chapter/10.1007/978-3-642-36844-8_1

If you would also like to use DEX to power your research do not hesitate to contact us and request a free research license to join the research program!

What?

We would like to share an interesting article from the RMIT University in collaboration with the CA Labs from CA Technologies about the detection of threats of insiders. DEX graph database is used as the management system to power their analysis.

Insiders are those people who work, or have previously worked, in a company and intentionally misused the access to compromise some information available. A popular example is Wikileaks, and how the threat of insiders should be a concern for any company. Nowadays, with the outsourcing done with the “cloud computing”, it is more important to detect insider attacks than ever .

With this issue in mind, the researchers at RMIT and CA labs want to propose an analysis in order to detect deviations of users from normal behavior while accessing the systems, using DEX graph database in order to benefit from its capabilities to store huge volumes of data to be analyzed.

How?

From 3 years of logs (2008 to 2011) extracted from the SVN access of a certain CA program they obtained 700M lines of access logs, and 282 unique users. In order to deal with such huge numbers they chose DEX graph database management system, which allowed them to store the following databases:

• Log database, with 700M nodes and 3500M edges, a really huge database with a total size of 305GB
• Command database, storing the commands executed by the users accessing the SVN. This is a smaller database of 6GB total size

DEX graph databases were used in the cluster analysis to detect communities, based on the accessed resources, projects and the daily access patterns. They discovered that a deviation on the daily pattern can be an alert of a possible insider threat.

For more details about the analysis, conclusions and future work we recommend reading the complete article here.

Our congratulations to the researchers at the RMIT University & CA Labs for such an interesting investigation towards building more secure systems for companies.

If you are also interested in using DEX for your research, do not hesitate to join the research program!

—–

CA Labs was established in 2005 to strengthen relationships between research communities and CA Technologies. CA Labs works closely with universities, professional associations and government organizations on various projects that relate to our company’s products, technologies and methodologies. The results of these projects vary from research publications, to best practices, to new directions for products.

BMAT trusts Sparsity Technologies solutions to improve its matching engine so as to be able to identify, with the highest quality, all the music available worldwide.

BMAT listens, analyzes and processes more than 2000 radio and television broadcasters from around the world to generate precise music playlist information for every monitored channel. For BMAT, to have a duplicate-free, high-performance and reliable matching engine that is able to identify all the music in the world is key to providing a meaningful and trustworthy monitoring service for their clients.

Sparsity Technologies enables this by offering BMAT its Extreme Data solutions. Sparsity’s Graph Databases can manage and analyze billions of data points in real-time, providing an unprecedented level of performance as demonstrated in the best graph benchmarks in the Industry*.

Song titles, artists, collaborations, composers and all the metadata related with the music industry can be conveniently represented as a graph. BMAT uses DEX as the core for their matching engine where hundreds of millions of entities and relations are stored, indexed, processed and queried efficiently. BMAT includes music intrinsic characteristics (such as mood, tempo, rhythm, style, etc.) to each song to further improve the quality of the identifications.

Sparsity’s solutions are the only ones capable of this level of performance on commodity hardware. Traditional solutions, even modern Big Data solutions could just not cope with the complexity involved to support this level of analysis in near real-time, especially considering the combination of music metadata information and music characteristics.

Sparsity Technologies is proud to count with BMAT as a client of its Big Data solution DEX.

About BMAT

BMAT (Barcelona Music & Audio Technologies) was founded in February 2006 as a spin-off of MTG (Music Technology Group) of UPF (la Universitat Pompeu Fabra). MTG is the world-renowned research centre dedicated to developing audio and digital music technologies. From its headquarters in Barcelona, BMAT collaborates with more than 50 clients worldwide; clients include Samsung, Yamaha, Intel, Telefónica, EMI and Endemol. BMAT is also fastest-growing provider of music monitoring service of radios and televisions in the world, servicing more than 30 performing rights organizations and collecting societies globally.
For more information, please visit www.bmat.com

SNTalent leverages Sparsity Technologies solutions to scale recruiting to hundreds of millions of CVs and job offers.

SNTalent and Sparsity Technologies have entered into an agreement whereby SNTalent will use Sparsity solutions and development platform to provide a world-class recruiting marketplace capable of scaling to 10’s of millions of CVs and job offers.

SNTalent has always been a pioneer in using technology as a key differentiator in the recruiting space, with solutions capable of searching candidates in middle management, technical and senior professionals, and matching these with open reqs. SNTalent now wants to scale massively to 10’s of millions of CVs on a global basis and has been looking for the ideal Big Data solution to support its core business.

Sparsity Technologies offers Extreme Data solutions. Sparsity’s Graph Databases can manage and analyze billions of data points in real-time, providing an unprecedented level of performance as demonstrated in the best graph benchmarks in the Industry*.

SNTalent has defined a very strong methodology for managing the lifecycle of job fulfilling. Key to this methodology is the shortlisting process, whereby a list of millions of CVs is dynamically reduced to the best 10 or 20 candidates based on customer specific criteria, SNTalent’s recruiting know-how, and relationship data. Allowing 1000’s of concurrent shortlistings in near real-time on 10’s of millions of CVs is clearly what positions SNTalent as the preferred marketplace and platform for recruiting globally.

Sparsity’s solutions are the only ones capable of this level of performance on commodity hardware. Traditional solutions, even modern Big Data solutions could just not cope with the complexity involved to support this level of analysis in near real-time, especially considering feedback loops from previous customer engagements, and relationships between candidates, sponsors and customers – typical requirements in the area of “social analysis”.

Sparsity Technologies is proud to have been selected by SNTalent as the provider of its Big Data solution.

* (Benchmarks are available at http://sparsity-technologies.com/blog/?p=196 and http://sparsity-technologies.com/blog/?p=228)

About SNTalent
SNTalent is a B2B company with headquarters in Spain, Germany and Brazil. We combine search technology in social media with a Marketplace of Recruitment 2.0 certified consultants, experts in attracting and selecting middle management, technical and senior professionals.

In this second release of the series of the use cases, we are looking through one of the most interesting scenarios for graph databases: Social Network Analysis (SNA).

DEX highest-performance with huge volumes of processed data, its flexibility and the nature of the graph, makes it the perfect solution for Social Network Analysis.

More info? We have created a new section in DEX site called Scenarios that contains a detailed explanation about the fields where Graph Databases are key, and which we have plenty of experience. We will be adding more, stay tuned!

Do not forget to check the list of features we believe SNA must cover. We welcome your feedback! Please tell us which do you think are SNA requirements and achievements and why graph databases could be a good solution.

If you think SNA is your area, we encourage you to evaluate DEX here, and do not hesitate to contact us at info@sparsity-technologies.com for additional support. Use our knowledge in the SNA field!.

Read also the first release of the Use Case series: Bibliographic exploration

Sparsity Technologies announces its new offices opening this April at the K2M (Knowledge to Market) building.

UPC Park was conceived with the mission to become a socioeconomic dynamic between UPC, administration and companies in order to promote research, innovation and transfer of technological progress and results.

You can now find us at floor 0 (hall level) offices 001a.

APTE also shares the news here: http://www.apte.org/es/noticia-parque529.cfm

The latest version of DEX includes the helpful algorithm package that give more high-level operations to the API.

Here you can find the list of algorithms explained including examples of use for the Java and .NET APIs:

Traversals algorithms
To traverse a graph is to visit the nodes included in the graph. You can choose between DFS or BFS techniques.

DFS (depth first search) is a technique were the nodes are visited starting at the root and selecting one of the neighbors’ nodes which are explored as far as possible along each branch before backtracking.

BFS (breadth first search) is a technique were the nodes are visited starting at the root which all its neighbors are explored and so on.

For both techniques you can restrict the visit by a certain type of nodes or only navigating through a certain type of edges.

Java example:

System.out.println("Traversal BFS");
// Create a new BFS traversal from the node "startingNode"
TraversalBFS bfs = new TraversalBFS(sess, startingNode);
// Allow the use of all the node types
bfs.addAllNodeTypes();
// Allow the use of all the edge types but only in outgoing direction
bfs.addAllEdgeTypes(EdgesDirection.Outgoing);
// Limit the depth to 3 hops from the starting node
bfs.setMaximumHops(3);
// Get the nodes
while (bfs.hasNext())
{
long nodeid = bfs.next();
int depth = bfs.getCurrentDepth();
System.out.println("Node "+nodeid+" at depth "+depth+".");
}
// Close the traversal
bfs.close();


The same with the TraversalDFS method.

.Net example:

System.Console.WriteLine("Traversal BFS");
// Create a new BFS traversal from the node "startingNode"
TraversalBFS bfs = new TraversalBFS(sess, startingNode);
// Allow the use of all the node types
bfs.AddAllNodeTypes();
// Allow the use of all the edge types but only in outgoing direction
bfs.AddAllEdgeTypes(EdgesDirection.Outgoing);
// Limit the depth to 3 hops from the starting node
bfs.SetMaximumHops(3);
// Get the nodes
while (bfs.HasNext())
{
long nodeid = bfs.Next();
int depth = bfs.GetCurrentDepth();
System.Console.WriteLine("Node "+nodeid+" at depth "+depth+".");
}
// Close the traversal
bfs.Close();


The same with the TraversalDFS method.

Find shortest path algorithms
Find the shortest way to travel from one node to another. The APIs offer two techniques BFS or Dijkstra. Dijkstra is the one to use if you have weights, that matter in the path retrieval, in the edges whileas BFS is the one to use otherwise.

Java example:

System.out.println("SinglePairShortestPath BFS");
// Create a new unweighted shortest path from "startingNode" to "endingNode"
SinglePairShortestPathBFS spBFS = new SinglePairShortestPathBFS(sess, startingNode, endingNode);
// Allow the use of all the edge types in Any direction
spBFS.addAllEdgeTypes(EdgesDirection.Any);
// Allow the use of all the node types
spBFS.addAllNodeTypes();
// Calculate the shortest path
spBFS.run();
// Check the path if it exists
if (spBFS.exists())
{
// Get the total path cost
System.out.println("A shortest path exists with cost: "+spBFS.getCost()+".");
// Get the path
OIDList pathAsNodes = spBFS.getPathAsNodes();
OIDListIterator pathIt = pathAsNodes.iterator();
while (pathIt.hasNext())
{
long nodeid = pathIt.next();
System.out.println("Node: "+nodeid);
}
}
else
{
System.out.println("No path found");
}
// Close the shortest path
spBFS.close();


Analogously the Dijkstra method.

.Net example:

System.Console.WriteLine("SinglePairShortestPath BFS");
// Create a new unweighted shortest path from "startingNode" to "endingNode"
SinglePairShortestPathBFS spBFS = new SinglePairShortestPathBFS(sess, startingNode, endingNode);
// Allow the use of all the edge types in Any direction
spBFS.AddAllEdgeTypes(EdgesDirection.Any);
// Allow the use of all the node types
spBFS.AddAllNodeTypes();
// Calculate the shortest path
spBFS.Run();
// Check the path if it exists
if (spBFS.Exists())
{
// Get the total path cost
System.Console.WriteLine("A shortest path exists with cost: "+spBFS.GetCost()+".");
// Get the path
OIDList pathAsNodes = spBFS.GetPathAsNodes();
OIDListIterator pathIt = pathAsNodes.Iterator();
while (pathIt.HasNext())
{
long nodeid = pathIt.Next();
System.Console.WriteLine("Node: "+nodeid);
}
}
else
{
System.Console.WriteLine("No path found");
}
// Close the shortest path
spBFS.Close();


Analogously the Dijkstra method.

Connected components algorithms
Connectivity shows in which degree a group of nodes are connected to each other. With DEX you can find strongy connected components using Gabow technique or weakly connected components using DFS technique.

Java example:

System.out.println("Weak Connectivity DFS");
// Create a new WeakConnectivityDFS
WeakConnectivityDFS weakConnDFS = new WeakConnectivityDFS(sess);
// Allow the user of all the edge types
weakConnDFS.addAllEdgeTypes();
// Allow the use of all the node types
weakConnDFS.addAllNodeTypes();
// Don't set a materialized attribute
// Calculate the weakly connected components
weakConnDFS.run();
// Get the connected components
ConnectedComponents weakCC = weakConnDFS.getConnectedComponents();
long numWeakComponents = weakCC.getCount();
System.out.println("Weakly connnected componennts: "+numWeakComponents);
for (long ii = 0; ii < weakCC.getCount(); ii++)
{
Objects ccNodes = weakCC.getNodes(ii);
long numNodes = ccNodes.count();
System.out.println("Connected component "+ii+" has "+numNodes+" nodes.");
ccNodes.close();
}
// Close the connected components
weakCC.close();
// Close the WeakConnectivityDFS
weakConnDFS.close();


Analogously the StrongConnectivityGabow method.

.Net example:

System.Console.WriteLine("Weak Connectivity DFS");
// Create a new WeakConnectivityDFS
WeakConnectivityDFS weakConnDFS = new WeakConnectivityDFS(sess);
// Allow the user of all the edge types
weakConnDFS.AddAllEdgeTypes();
// Allow the use of all the node types
weakConnDFS.AddAllNodeTypes();
// Don't set a materialized attribute
// Calculate the weakly connected components
weakConnDFS.Run();
// Get the connected components
ConnectedComponents weakCC = weakConnDFS.GetConnectedComponents();
long numWeakComponents = weakCC.GetCount();
System.Console.WriteLine("Weakly connnected componennts: "+numWeakComponents);
for (long ii = 0; ii < weakCC.GetCount(); ii++)
{
Objects ccNodes = weakCC.GetNodes(ii);
long numNodes = ccNodes.Count();
System.Console.WriteLine("Connected component "+ii+" has "+numNodes+" nodes.");
ccNodes.Close();
}
// Close the connected components
weakCC.Close();
// Close the WeakConnectivityDFS
weakConnDFS.Close();


Analogously the StrongConnectivityGabow method.

Finally do not forget to include the package when using the former methods! by adding:

Java:
import com.sparsity.dex.algorithms.*;

.Net:
using com.sparsity.dex.algorithms;

Bibliographic exploration is an interesting use case for Graph Databases. Bibliographic exploration rises after the need to query huge bibliographic resources to obtain relevant information for researchers.

There are many questions that researchers try to ask to Bibliographic resources, but the vast amount of heterogeneous information stored in them makes it difficult to obtain good and fast answers.

Articles, its authors and the keywords that best describe those articles are stored in Bibliographic resources. This type of information is naturally linked, for instance authors are linked with other authors by the articles they have collaboratively written and at the same time articles may be connected with the keywords that are most relevant in them.

Graph Databases are a good solution to store huge amount of strongly connected information. Graph Databases store information the same way it is connected naturally; therefore answers can be retrieved directly without having to join all the data as it happens in SQL traditional databases.

The following figure is an example of how a Bibliographic resource may be stored in a Graph Database.

Capture from Bibex online demo

We can see in the illustration that authors are nodes in the graph, and they are connected by their collaboration in papers (edge). With a click on the edge of the graph you can obtain all the articles written together by both authors.

Capture from Bibex online demo

This type of query takes seconds to have a result in a Graph Database and could be relevant to new researchers, like PhD students, or researchers in a new area in order to investigate authors, the papers they have written, who they have collaborated with and about what topic areas.

If you want to play with this type of query in a graph, visit Bibex social network free demo. Bibex is able to process large quantities of data because it is powered by DEX Graph Database. The online demo stores the information from DBLP a well-known bibliographic resource, but it could use any other bibliographic source, even combining them.

Another interesting aspect about storing Bibliographic information with Graph Databases is the use of the citation metric. An article or an author can be considered to be of quality depending on both the number and the acknowledgment (quality) of the citations. Again Graph Databases are the most suitable technology to work with this metrics, since it would represent only retrieving the neighbors for a certain node “article”, that have the edge type “cite”:

//Once the DB is open

//You should close here the DB
Example of code using DEX Graph Database Java API

Using citations we could answer questions like “Who is an authority in a certain topic?” or “Who is the most suitable reviewer for a certain paper?” The possibility to answer those new complex queries is what makes graph databases an excellent use case for bibliographic exploration.

Let’s conclude with the big pros of using Graph Databases for Bibliographic exploration:

• Data sources with bibliographic information are huge and strongly connected. Graph Databases can store billions of objects and are specially created to store linked information.
• Bibliographic exploration is more interesting if it merges as many sources as possible. Graph Databases can store data with heterogeneous schemas, like bibliographic repositories, publishers, patents, or any other source of information.
• Researchers need to have answers as quick as possible, in order to have his/her efforts focused in its main topic of research. Graph Databases can query connected data in a few seconds, even for complex queries.
• New complex questions can be easily answered using graph database ease to navigate through linked information.

We would like to announce the release of a complete free demo for the Bibex social network query.

Click here to launch Bibex demo

Bibex resolves multiples queries that help retrieving very relevant information for researchers in very short responding times. Bibex uses DEX graph database to resolve questions like “who is the most important authority in a certain subject?” in a few seconds. Moreover as Bibex results are shown in a network it is easy to jump between articles, keywords and authors while navigating the answer. Read more information about Bibex here .

Bibex demo is able to resolve the “Social network” query available in Bibex.

You will be able to search for the social network of any author*, retrieve their curriculum, relationships & statistics. In addition you can jump to the publication source of each article with just one click. Check how Bibex resolves this query in our Bibex demo here.

Bibex social network has an intuitive web interface. Once you click on this link, a search box will be shown. There you can search for any author name you think of*. For instance, in the following example we search for Tim Berners-Lee. Search box has an autocomplete facility to help you discern faster the exact author you are trying to search.

Once the search is performed, if there’s only one possible answer to your query the author curriculum and its social network are shown right away. If there are multiple answers a list of possible authors appear on the left in alphabetical order with the first author social network already loaded in the right side of the screen.

In that second case you must click on the name you were actually searching for to load the social network of the author.

Authors’ curriculum on the left contains a list of all their publications. They can be sorted by date, alphabetically or you can search inside the list as well. Selecting a publication and clicking in the “Go to URL” icon shown in the following picture jumps you to the original source of the publication.

Authors’ collaborations on the left contain a list of all the authors that have co-written some publication. It is interesting to sort this list by number of collaborations allowing discerning which have most strong relationships and therefore may be also of your interest.

Finally on the bottom of this left side there’s a statistic of the productivity of the author through time.

Another important part of the results are shown in the right side of the screen. The social network of the author can be navigated; discovering all the relationships.

Double clicking on another author’s name jumps you to that author information and social network. In addition, clicking two authors’ edge reveals a list of the publications written collaboratively by both authors.

Hope you enjoy Bibex, feel free to play and experience the smoothness and quickness of answer.

*Bibex demo searches in DBLP bibliographic database. The DBLP used for Bibex has an amount of 999,053 authors and 2,740,244 articles.

The possibilities of native .NET programming now for the highest performance graph database.

Now you can have all the scalability and performance of DEX graph database with a dedicated Microsoft .NET API in your secure and robust professional MS environment.

DEX new release comes with a completely renovated Java API and the brand new .NET API, for .NET languages programmers.

DEX does not forget compatibility with applications that use previous versions of its API, that’s why we not only give an easy migration guide but also offer an API for DEX 4.2 with compatibility with previous versions. Nevertheless, we recommend to all DEX programmers to start migrating to the new Java API, since we assure the process to be quite painless and quick and will guarantee the continuity of all their applications.

We are proud to announce new Daurum 5.0 version of our software.

Daurum 5.0, the deduplication and integration tool is able to integrate data from multiple sources linking millions of records and identifying potential duplicated records.  A remarkable characteristic of this new version is its friendly web interface that allows concurrent access by multiple users.  With Daurum 5.0 users may have different roles sharing their data and, most important, dividing tasks between them.  Also, Daurum 5.0 allows users to create their own filters for data cleansing, using an integrated editor with preview options, or choose one from the list of the tested and most valuable default filters.

Daurum 5.0 is coming with new exciting possibilities. Now you can test our deduplication technology with an online demo, request our services, or buy a license of the software.

Daurum online service demo is a free service we offer through our website to take a first glimpse of the results that can be achieved with Daurum 5.0 services or software. Test one typical scenario for the client database and see how duplicates are detected. Please read through our description of the demo, and the comparative table between what you can actually find in Daurum that is not available in the demo service.

We hope that the online service demo becomes a useful evaluation tool in order to understand how deduplication works.

We offer Daurum 5.0 in a 2 brand new modalities:

If you need to clean your database of duplicated records, or if you have several databases to be merged assuring the best quality for the resulting database, consider hiring Daurum services.

Daurum services prices are tailored to your database size and cleansing needs, it is always the best option if you need one shot deduplication.

If you want your own copy of Daurum 5.0 for perpetual use, we would recommend one of our licenses.

With Daurum license, you will be able to deduplicate or merge databases as many times as you need. Create your own specific filters for the detection of duplicates, add your dictionaries and become the best expert in your database.

Also the license includes a professional installation support.

To see a little more about the potential of Daurum 5.0 services & software, with its complete functionalities, take a look at the following images:

We would like to congratulate DAMA-UPC that has just been awarded with the Bdgital congress prize in the category of Research institutes and Universities.

RECOMANA* uses the potential of graph databases with the high-performance and billions of objects storage offered by DEX to analyze different type of information from media content to school programs, in order to give a tool to the professionals of education to obtain better and specific information for teaching purposes.

The quantity of audiovisual content generated by media such as TV broadcasters and Radio stations is huge and forms a significant body of knowledge that spans many areas. However, this content is not available to the professionals of education in an efficient way. In other words, teachers have to devote a significant amount of their times to obtain videos or fragments of videos useful for their teaching purposes, without knowing whether they are accurate for their objective.

The final objective is to allow society have an asset and benefit from RECOMANA project, allowing teachers and students to access digital information to improve the contents of the topics in primary and secondary education.

BDigital congress new: http://www.bdigitalglobalcongress.com/lang-en/2011/zyncro-invoxcontact-recomana-and-webdom-enlight-winners-of-the-prizes-awarded-at-the-bdigital-global-congress/

RECOMANA project video (in catalan):  http://www.youtube.com/user/BarcelonaDigital#p/c/20/jar83ywfEf0

*RECOMANA in Catalan means recommend, suggest

DEX has a new collaborative support area to share doubts, suggestions or any request about DEX. We encourage your participation in the new online forum callled DEX technical support area. We would like the DEX technical support area to be the meeting point for all DEX users. Share your doubts with our technical stuff and use your expertise in DEX to help other users.

To enter to the area, please click on the main page and enter in the technical area using the button shown in the following images:

Also you can use DEX subsection Request info to access to the technical area.

Once you are in the Technical Area a list of the available forums will be shown. Click on the DEX Technical Support Forum to see the list of current topics.

If you are new to DEX we highly recommend to check the always updated complete list of the most frequently asked questions (FAQ), which you will find in the top of the forum topics list. New questions from your requests in the forum will be included in the FAQ . Do not forget to check the list, you may find your answers there!

Now you can choose to read our website in Catalan or Spanish. All contents are accessible in the new multi-language platform. To change your language just search it in the top of the page:

Yet we are keeping the blog only in English. Please feel free to share with us your impressions or any suggestion!

Sergio Gómez from Sparsity Technologies presented DEX last Saturday 16th of April at the GDM 2011 in Hannover. The workshop included research and industrial sessions.

DEX presentation includes new interesting details about the internal technology and architecture of the DEX graph database .

We would like to share one of the latest use cases we have recently performed at Sparsity Technologies with DEX.

We have created a DEX graph loading all tweets made from June to December 2009, with a total of more than 476M tweets and 1.2 billion follower relations.

The resulting graph has 655M nodes belonging to 4 different types: users, tweet, hashtag and url. Former nodes have relations such as retweet, follows or references making a total of 2.2 billion edges present in the graph. Nodes and edges have a total of 1.6 billion attributes. Twitter objects make a final 4.5 billion DEX graph database.

See the resulting schema:

The loading was made in a Linux machine with 64GB RAM and a single CPU and the resulting graph database has 192GB; this is 3 times the available memory.

Resulting graph is ready to be queried. If you were wondering if social networks such as twitter or facebook could take advantage of graph databases, we are positive this test contributes to the exploration of this interesting scenario.

Questions like “Is it reasonable to work with such a big graph?” or “How long queries to the graph will take?” arise. We’ll return with more information.

Sparsity Technologies is proud to announce the new release 4.1 of DEX.  The new version of DEX brings extended compatibility with common operating systems, to former versions of the product.

In addition to DEX 4.0, new subversion includes compatibility with MAC OS Snow Leopard and Windows 64 bits.  DEX 4.1 offers an export to graph standard GraphML, a multi-level log system and more efficient use of space, among other improved characteristics.

Start using DEX 4.1 now with our free personal evaluation license or contact us for any license that that fits you better.

Together with the people from tinkerpop we have today released the DEX implementation of the Blueprints API.

“Blueprints is a property graph model interface that provides implementations, test suites, and supporting extensions. Graph databases and frameworks that implement the Blueprints interfaces automatically support Blueprints-enabled applications. Likewise, Blueprints-enabled applications can plug-and-play different Blueprints-enabled graph backends.”

DEX natively supports the property graph data model defined by Blueprints. See more information at: https://github.com/tinkerpop/blueprints/wiki/Dex-Implementation

Let us know your impressions!.

We would like to thank Marko A. Rodriguez and the people at Tinkerpop for their support.

We would like to thank CeBIT organizers and in particular the Spanish pavillion for their work in this last edition of CeBIT 2011 in Hannover.

CeBIT 2011 was a total sucess with 4200 exhibitors from 70 different countries and 339.000 vistors from 90 different countries. Ernst Raue, responsible of CEBIT, states that CeBIT has become an international reference for stablishing business contacts, with 7 millions of commercial contacts made in 45 hours.

Sparsity Technologies in its first attendance to CeBIT would like to thank the large number of visitants that stopped by our stand and were interested in our solutions. We had the chance to meet lots of contacts from diverse and very interesting fields.

Next CeBIT will take place in Hannover from the 6th to 10th of March 2012.

Sparsity Technologies altogether with DAMA-UPC  are now at CeBIT 2011 for the whole week. Starting yesterday and until Friday 5th we will be glad to receieve you at our Stand A06 at Hall 09 with the latest information about DEX 4.0, Daurum 5.0 and Bibex. If you are in Hannover do not forget to stop by.

Here are some pictures from the very first day before the opening:

The First University-Industry Meeting on Graph Databases (UIM-GDB) is an initiative to gather companies developing graph database management systems together with researchers from academia. The main objective is to join efforts to overcome some of the main obstacles in the development and acceptance of graph databases in industry.

The meeting hosted by DAMA-UPC and Kobrix started today and will take place at the Universitat Politècnica de Catalunya, in Barcelona. Barcelona (Catalonia, Spain).

Main objectives for the meeting are:

• Present different graph database management systems in the market.
• Discuss customer needs regarding graph-like data management and analyze the most common query types.
• Foster the standardization of a query language on generic graphs.
• Create a collaborative forum to exchange ideas related to graph management and foster research in this area.

Schedule and attendees:

More information at: http://www.dama.upc.edu/research-2/UIM-GDB

As we have already announced in the previous post we would like to share an analytical use case that shows DEX high performance. This time we are taking a look at how DEX responds to some queries performed on a single dataset taking as a reference the results of another well-known open source graph database.

With this benchmark we would like to join the celebration of Wikipedia for its 10^th anniversary. Wikipedia was launched in 2001 by Jimmy Wales and Larry Sanger and has become the largest and most popular general reference work on the Internet having 365 million readers. Our congratulations to everyone making Wikipedia possible!

For the benchmark we used all the Wikipedia articles written before January 2010. In particular the database loaded contained 55M articles, 2.1M images and 321M references between articles.

With this benchmark we want to obtain the following information:

1. Loading times, including the generation of full index structures for the graph.
2. Graph database size
3. Response times for typical queries made to the loaded data, which include:
   1. Query 1(Q1): Finds the node with the maximum outdegree, the one with most relationships with other nodes, and then runs a BFS traversal of the graph starting from that node. More info about traversals algorithm in the graph algorithms post.
   2. Query 2(Q2): Finds the node with the maximum indegree, select nodes referencing that node, and with this new set, finds again those referencing every node in the set. In other words, it performs the 2-hops operation. Finally the query ranks the nodes by number of references and returns de top 5. 
   3. Query 3(Q3): Finds a pattern in the graph.  The pattern tries to find articles written in Catalan (CA) which are translated into English (EN) without some images from the original article.
   4. Query 4(Q4): Finds the number of articles and images for every language available.
   5. Query 5(Q5): Materializes the number of images for all the articles.
   6. Query 6(Q6): Deletes all the articles from the database with no images.

See the results in the following table:

It is remarkable to notice that loading all the articles from the Wikipedia to DEX only took 2.25 hours with a resulting database size of 16.98 GB. This shows the huge amounts of information that can be loaded to DEX with no disk restrictions in reasonable times.  The results for all six queries are always positive for DEX with results of more than two orders of magnitude for all the queries except for Q3, which still is one order of magnitude faster.

DEX gives the greatest performance results in both loading time and responding time to queries, making DEX an attractive option for those solutions with big volumes of data that are cumbersome to analyze.  Try DEX now and you’ll see how this performance is in action.

With this post we would like to show you some performance results that will explain why DEX is claimed to be a high-performance graph database with great scalability.

To go further into DEX scalability let us see the results using the R-MAT benchmark testing DEX performance for very large databases. In this benchmark we are interested intro tracking the following indicators:

• How many nodes and edges could be created?
• Which was the size of the database created?
• How long did the load of the database take?
• How many traversals we could possibly make per unit of time?

To test the scalability we automatically created some graphs with 2^SF nodes (for a scale factor, SF, ranging from 25 to 28) and the number of edges (shown in column 3 of the table) automatically created by the R-MAT synthetic graphs generator.  The rest of the columns depict the size, load time and execution measures for a heavy traversal query.

Load time and size of a graph

For this benchmark we load up to 2.1 billion edges and 230 million nodes.  For this very large graph, answers to the former questions are quite impressive: the graph was loaded in only 15 hours, including the creation of all the indices for the direct access to the nodes and edges. The edges are loaded at a rate of 40K per second.  This large database occupies 83 GB , which leads to an average of only 36 bytes per node or edge.

Traversal query

In addition, we made a query against the database to test its response time (Q1 in the table).  Query 1 founds the node with the maximum out degree and then, it performs a BFS traversal starting from the node selected.   We can see that 4.2 billion traversals are made with an average of 295K nodes traversed per second.  More information about the BFS algorithm can be found on the post about DEX graph algorithms.

We obtained the remarkable results shown in the third rightmost column, with a minimum of 24 minutes for the SF=25 graph, and 4 hours for the SF=28 graph, with a degradation of less than 5% between scale factors.

Conclusions

Considering its fast loading and querying DEX is the graph database to go when dealing with large datasets because it has great performance for graphs with billions of objects.  Furthermore, it does not only give quick answers but also its size is optimized with only 36 bytes per object available in the database.

Stay tuned for more benchmarks to come, including an interesting analytical use case with Wikipedia.

Note: The experiments are performed using a computer with two quadcore  Intel(R) Xeon(R) E5440 at 2.83 GHz. The memory hierarchy is organized as follows: 6144 KB second level cache, a 64 GB main memory and a Disk with 1.7 TB. The operating system is Linux Debian etch 4.0.

UPDATE:

We have uploaded the code of the R-MAT syntethics graph generator we implemented for this benchmark in the download section of DEX . With this code the same benchmark can be repeated for other graph databases. We also include the code for DEX from the query 1 tested in the benchmark. To test this query for another graph database it should be adapted first. Please feel free to download it and share with us your results!

CeBIT is the digital industry’s biggest, most international event. Thanks to its unique combination of exhibition, conferences, keynotes, corporate events and lounges, CeBIT represents an unrivaled tool for doing business and sealing deals.

CeBIT will be held in Hannover from the 1st to the 5th of March. Sparsity Technologies and DAMA-UPC will be together at Stand A06 from Hall 09. Please do not forget to stop by.

Sparsity Technologies is proud to announce the new release 4.0 of DEX.  The new version of DEX brings extended performance to former versions of the product.  In addition DEX 4.0 brings automatic system configuration, creating an automated optimum configuration without any DBA effort.  DEX 4.0 also has cross-platform compatibility even between 32 and 64 bits platforms.  Start using DEX now with our free personal evaluation license or contact us for any license that that fits you better.

A DEX webinar will be held in mid-January 2011 including more details of this new release.  Please stay tuned for the webinar date announcement.

More information at www.sparsity-technologies.com or email us at: info@sparsity-technologies.com

This is the last feature we are posting on the blog for DEX 3.0. This time we are taking a look at the types allowed by graph databases for its objects.

String type only allows a maximum length of 2048 characters but you may need to create an attribute with more information than that. Take for instance a graph with academic articles, a typical attribute could be its abstract but it doesn’t fit in a string. In DEX 3.0 there’s a new type called TEXT that fits these attributes requirements with an unlimited number of characters capability. TEXT is managed as a stream instead of using the set and gets methods used in regular types, thus a channel is created to read and write iteratively until it is closed by the user.

Hope you find this new type useful. Do not forget to check latest posts in case you missed them: graph algorithms, regular expressions, neighbors materialized or indexed or not indexed attributes.

Fourth episode of the series about DEX features. DEX is all about performance, with this feature we are focusing on improving query performance.

In the DEX graph every node and edge can have as many attributes as needed. Those attributes were automatically indexed in order to improve query performance, but it also had a negative turn in means of space cost and loading time. However not every attribute is used in queries so why do we have to pay a performance penalty for attributes that will never be queried? In DEX 3.0 it’s the developers’ choice which attributes should be indexed and which ones are not, improving substantially the global performance. Use public long newAttribute(int type,java.lang.String name, short datatype, short kind) and choose among the following attributes types for “kind”:

• Graph.ATTR.KIND_BASIC: allows storing a value for each object but they can’t be used at query methods.
• Graph.ATTR.KIND_INDEXED: attributes will be indexed for querying.
• Graph.ATTR.KIND_UNIQUE: attributes are indexed and adds a restriction to not accept objects with the same value (only null values can be duplicated).

This feature makes developers more aware of performance when creating objects for their graph optimizing where it’s more meaningful.

Hope you find this issue interesting. Do not forget to check latest posts in case you missed them: graph algorithms, regular expressions or neighbors materialized.

One common challenge when traversing a graph is to obtain the neighbors for a certain node. In previous versions of DEX it required a two-step operation; first obtaining the edges for the node and then its neighbors.

Objects neighbors = new Objects(graphPool);
Objects e = graph.explode(node, edgeTypeId, Graph.EDGES_OUT); //With “ explode” we obtain the outing edges of node
Objects.Iterator it = e.iterator(); //With the iterator we obtain all the neighbors for all the edges exploded
while (it.hasNext()) {
long eid = it.next();
long neighbor = graph.getEdgePeer(eid, node);
neighbors.add(neighbor);
}
it.close();


In DEX 3.0 if the user creates a specific index for this operation, the retrieval of neighbors is straight forward and therefore executed faster making it in just one step. The user when creating the edge must specify if neighbors must be materialized or not:

public int newEdgeType(java.lang.String name,boolean directed, boolean neighbors) //Where neighbors is “ true” if they must be materialized and “false” otherwise

And then for the retrieval operation only the method public Objects neighbors(long oid, int type, short direction) has to be used.

Hope you find this useful, stay tuned for more features to follow. Also do not forget to check graph algorithms and regular expressions posts.

In the second delivery of this series we are going to see how DEX extends the capability to query the graph database using regular expressions. As in every database you can also query the entities in your graph database with DEX. DEX 3.0 has enriched those queries by using regular expressions. Not only you can use operators:

• == (equal)
• != (not equal)
• > (greater than)
• >= (greater or equal than)
• <= (lesser or equal than)
• LIKE
• ILIKE (Like without being case-sensitive)

But also enrich your queries using regular expressions. Regular expressions provide flexibility to match strings with text, using a formal language with alphabet characters and new operations, such as:

• * (any character or expression or nothing else)
• + (at least one more character)
• | (two string alternatively)
• ? (the following character must appear only once or never)
• See more implemented operators at: http://www.sparsity-technologies.com/downloads/javadoc/edu/upc/dama/dex/core/Graph.html#OPERATION_ERE

Regular expressions widen the possibilities to find the string you are precisely looking for and conform most of the POSIX Extended Regular Expressions. Let’s see how it can change a query, taking for instance the following example:

You are trying to search for “John Lee” among people stored in the database with the attribute “fullname”, but you are not sure whether the name is really “Lee” or ”Lea”, also you might be even more wrong and the “John” you are looking for is really “Light” or ”Lane”. Method  graph.select(fullname, LIKE, “John”) will not retrieve anything because there’s not any single person registered only by its name; the last name is also required, then if you try graph.select(fullname, LIKE “John Lee”) you will discover that you are misspelling his last name. Your search will normally end here, but with DEX 3.0 you can use regular expressions by selecting all “Johns” that have a last name that begins with an “L”: graph.select(fullname, ERE, "^John L.*") and you will obtain a graph with all the people that match that query. Then it’s your turn to decide which”John L” is the one. In just one single search you retrieve what you exactly were looking for.

Stay tuned for more features to come. Do not forget to check the first episode: graph algorithms

Starting today we are going to make a series of posts explaining the most exciting features of the last available version of DEX. We hope that these explanations will give you a hint on the powerful possibilities of DEX. If some curiosity is left we highly encourage you to try the free download of DEX at http://www.sparsity-technologies.com/dex_downloads.php and test all its features.

Let’s start with one of the basic features of DEX 3.0, the graph algorithms.

There is a large literature on graph algorithms studied in the theory of graphs that has been proven to give excellent results extracting information with graphs. DEX 3.0 library offers the following valuable algorithms:

• Traversal: You can move through the graph using BFS (breadth first search – starting at the root all its neighbors are explored and so on) or DFS (depth first search – starting at the root and selecting one node neighbors are explored as far as possible along each branch before backtracking).  The former methods allow to restrict which node types do you want to visit and which way are the edge types navigable. Use the traversal methods to obtain all the nodes in the graph ordered at your choice.BFS example:
  Graph DEXGraph -> Dex graph instance
  long root_node -> valid node identifier from the graph DEXGraph
  long  name -> existing attribute indentifier

  TraversalBFS t = new TraversalBFS(DEXgraph, root_node);
while(t.hasNext()){
long current_node = t.next();
string name = dbg.getAttribute(current_node, name);
system.out.println(name.toString());
}
t.close();

  DFS example: Same as for BFS using method  TraversalDFS t = TraversalDFS(DEXgraph, root_node);

  See extended examples at:  http://www.sparsity-technologies.com/downloads/javadoc/edu/upc/dama/dex/algorithms/package-summary.html#usage

• Shortest Path: Having two nodes you can automatically discover which one is the shortest path between them. DEX 3.0 computes the well-known Dijkstra shortest-path algorithm (if there are weights in the edges) and BFS (without weights). It is also possible to restrict which node types do you want to visit and which way are the edge types navigable. In addition also the maximum longitude for the shortest path can be given.  Use the shortest path to discover, for instance, the minimum chain of contacts between yourself and the responsible of the company you would like to work with or to discover the fastest way to go to that restaurant from your home.Dijkstra example:
  Graph DEXGraph -> Dex graph instance
  long source_node -> valid node identifier from the graph DEXGraph
  long destination_node ->valid node identifier from the graph DEXGraph

  SiglePairShortestPathDijkstra sp = new SinglePairShortestPathDijkstra(DEXgraph, source_node, destination_node);
sp.run();
double path_size = sp.getCost();
sp.close();

  BFS example: Same as for Dijkstra using method  TraversalDFS t = TraversalDFS(DEXgraph, root_node);

  See extended examples at: http://www.sparsity-technologies.com/downloads/javadoc/edu/upc/dama/dex/algorithms/package-summary.html#usage

• Connectivity:  Two algorithms, one for searching strongly connected components using Gabow algorithm and another one for weekly connected algorithms using DFS will help you measuring the connectivity between entities in the graph.  Connectivity among entities will give for instance a measure of the communication among the people in your office; if they are more connected information will flow more fluently and quickly.Graph DEXGraph -> Dex graph instanceStrongConnectivityGabow scc = new StrongConnectivityGabow(DEXGraph);
scc.run();
ConnectedComponents cc= scc.getConnectedComponents();
scc.close();
//Retrieving the number of connected components found.
long totalCCS = cc.getCount();


  See extended examples at:  http://www.sparsity-technologies.com/downloads/javadoc/edu/upc/dama/dex/algorithms/package-summary.html#usage

For more information about any of these features check the technical documentation at: http://www.sparsity-technologies.com/dex_tutorials.php#dex_javadoc

We have uploaded the tutorial presentation made during last DEX Seminar. In the tutorial you will be able to see a further insight on DEX  having a look at some representative API methods, and how to use them with illustrative examples.

The presentation is available at DEX documentation section. Visit: http://www.sparsity-technologies.com/dex_tutorials.php#dex_tutorial

If you are new to DEX do not forget to take a look first to the Introduction presentation as well.  Visit:  http://www.sparsity-technologies.com/dex_tutorials.php#dex_intro

Sparsity Technologies gives a talk about Bibex at the last Barcelona Biomedical Research Park seminar that takes place June 2010.

For more information visit:
BIBEX, high performance exploration for bibliographic databases
Speakers: JOSEP LLUIS LARRIBA PEY (DAMA-UPC) / PERE BALETA FERRER (SPARSITY TECHNOLOGIES)

BIBEX, the high performance bibliographic exploration tool, developed and commercialized by DAMA-UPC and his spin-out Sparsity Technologies, is finalist at the 12h BDigital Global Congress in the small companies and entrepreneurs DIGITAL INNOVATION AWARDS category.

Sparsity-Technologies (formally Sparsity, S.L.) is a new spin-out of the UPC (Universitat Politècnica de Catalunya) created on March 2010to commercialize and provide services based on the technologies developed at DAMA-UPC.

The team of Sparsity Technologies is formed at DAMA-UPC and has a wide experience of more than 11 years in the investigation of leadership areas of information technologies and in particular with the improvement of performance in the management of large volumes of data. Sparsity team combines the experience in university research with its 10-year-experience collaborating with partners in the industry. The team is a well-balanced group of engineers and doctors fully experienced in the development of applications for the industry and top-research oriented to the business.

Sparsity-Technologies is located at Barcelona, a city with international projection known for its institutional interest for encouraging top-quality research and development of technology.

DAMA-UPC with DEX (Data Network Exploration) project wins the prize for novel enterpreneurs awarded by CINC in the 9th edition of Concurs d’Idees de Negoci – ACC10.

DEX is a high-performance graph database to manage very large graphs or networks of data, allowing the integration and exploration of multiple data sources.

DAMA-UPC, the DAta MAnagement group at Polytechnic University of Catalonia (Universitat Politècnica de Catalunya – UPC)  is part of the Computer Architecture Department (DAC). The work of our group focusses on the research and technology transfer in topics related to the management and retrieval of large data volumes, information quality and data exploration.

The CINC prize was collected by the members of DAMA-UPC: Josep Lluís Larriba-Pey, Víctor Muntés Mulero, Norbert Martínez Bazán and Pere Baleta Ferrer, on Dec 16th 2009.