List of Listings

Chapter 3. Some fundamentals

Listing 3.1. helloworld.sbt

Listing 3.2. hellworld.scala

Chapter 4. GraphX Basics

Listing 4.1. Construct a graph as shown in figure 4.3

Listing 4.2. Retrieve the edges from the just-constructed graph

Listing 4.3. Get a triplet(s) version of the graph data

Listing 4.4. Add Boolean annotation to edges indicating a condition

Listing 4.5. Using aggregateMessages[]() to compute the out-degree of each vertex

Listing 4.6. RDD join() to match up VertexIds with vertex data

Listing 4.7. map() and swap() to clean up output

Listing 4.8. rightOuterJoin() instead of join() to pull in “forgotten” vertices

Listing 4.9. Option[]’s getOrElse() to clean up rightOuterJoin() output

Listing 4.10. Iterated (via recursion) Map/Reduce to find distance of furthest vertex

Listing 4.11. Round-trip persisting to and reading from file

Listing 4.12. Saving to a single file in HDFS using the Hadoop Java API

Listing 4.13. Command line commands to use jackson-module-scala in the REPL

Listing 4.14. Naïve approach to serialize to JSON

Listing 4.15. Better performing way to serialize/deserialize to/from JSON

Listing 4.16. Export to GEXF for Gephi visualization software

Listing 4.17. Generate a grid graph

Listing 4.18. Generate a star graph

Listing 4.19. Generate a log normal graph

Listing 4.20. Generate an R-MAT graph

Listing 4.21. Pregel to find distance of furthest vertex

Chapter 5. Built-in algorithms

Listing 5.1. Personalized PageRank to find the most important related paper

Listing 5.2. Triangle Counts on Slashdot friend and foe data

Listing 5.3. Invoking ShortestPaths

Listing 5.4. Invoking connectedComponents()

Listing 5.5. Find and list social circles

Listing 5.6. Invoking stronglyConnectedComponents()

Listing 5.7. Invoking LabelPropagation

Chapter 6. Other useful graph algorithms

Listing 6.1. Dijkstra Shortest Paths distance algorithm

Listing 6.2. Executing the Shortest Path distance algorithm

Listing 6.3. Dijkstra’s Shortest Path algorithm with breadcrumbs

Listing 6.4. Executing the Shortest Path algorithm that uses breadcrumbs

Listing 6.5. Travelling Salesman greedy algorithm

Listing 6.6. Minimum Spanning Tree

Listing 6.7. Building the distances graph

Chapter 7. Machine learning

Listing 7.1. Invoking SVDPlusPlus with the data from figure 7.2

Listing 7.2. pred() function and invoking it

Listing 7.3. Downloading and cleaning the Reuters wire news items

Listing 7.4. Running LDA on the Reuters wire news items

Listing 7.5. Constructing the training set graph of figure 7.8

Listing 7.6. Preparing the data for Logistic Regression

Listing 7.7. Training the Logistic Regression model

Listing 7.8. Measuring the performance of the Logistic Regression model

Listing 7.9. Constructing the test data graph and evaluating its performance

Listing 7.10. PIC.sbt for image segmentation

Listing 7.11. PIC.scala for image segmentation

Listing 7.12. Brute force K-Nearest Neighbors

Listing 7.13. Generate example data

Listing 7.14. Custom export (with layout) to Gephi .gexf for knnVertex-based graphs

Listing 7.15. Execute K-Nearest Neighbors on the example data and export to .gexf

Listing 7.16. Distributed, approximate K-Nearest Neighborhood graph generation

Listing 7.17. Semi-supervised learning label propagation

Listing 7.18. Prediction function to use the semi-supervised learned graph

Listing 7.19. Execute semi-supervised learning label propagation and use it to predict a class (label) for a particular (x,y) coordinate of (30.0,30.0)

Chapter 8. The missing algorithms

Listing 8.1. Delete isolated vertices: handy for after having called subgraph()

Listing 8.2. The removeSingletons function in action

Listing 8.3. Merge two graphs into one

Listing 8.4. Using mergeGraphs on the philosopher graphs

Listing 8.5. Read a tab-separated RDF file

Listing 8.6. Read a tab-separated RDF file, performance improved with IndexedRDD

Listing 8.7. pom.xml for listing 8.6

Listing 8.8. Train SVDPlusPlus model on “<exports>” edges from YAGO

Listing 8.9. Compute vr, the list of potentially missing exports for Canada

Listing 8.10. map()-friendly SVDPlusPlus prediction function and helper

Listing 8.11. Find most likely Canadian export not in Wikipedia

Listing 8.12. Global clustering coefficient

Listing 8.13. Example using the global clustering coefficient

Chapter 9. Performance and monitoring

Listing 9.1. Transformations that require multiple stages

Listing 9.2. A simple iterative graph update algorithm

Listing 9.3. A simple iterative graph update algorithm with checkpointing

Listing 9.4. Using Kryo

Chapter 10. Other languages and tools

Listing 10.1. pom.xml

Listing 10.2. EdgeCount.java

Listing 10.3. EdgeCount.java fragment converted to Java 8 lambdas

Listing 10.4. drawGraph in d3.js JavaScript and Scala

Listing 10.5. sjsslashdot.sbt

Listing 10.6. sjsslashdot.scala

Listing 10.7. Benchmarking Triangle Count in GraphFrames

Listing 10.8. readRdf() rewritten in GraphFrames

Listing 10.9. Finding the friends of the friends of Ann using the Cypher subset

Listing 10.10. Using Cypher and graph isomorphisms to find missing Wikipedia info