How to build project from https://github.com/apache/spark using sbt

[Pallavim54]

can anyone mention the steps which i need to follow for building spark application also commands to test the scala testcase.referring from https://github.com/apache/spark

I m getting build error

Stacktrace

[ERROR] Failed to execute goal on project spark-core_2.10: Could not resolve dependencies for project org.apache.spark:spark-core_2.10:jar:1.1.0-SNAPSHOT: Failed to collect dependencies at org.easymoc
k:easymockclassextension:jar:3.1: Failed to read artifact descriptor for org.easymock:easymockclassextension:jar:3.1: Could not transfer artifact org.easymock:easymockclassextension:pom:3.1 from/to ma
ven-repo (http://repo.maven.apache.org/maven2): Access denied to: http://repo.maven.apache.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom , ReasonPhrase:Forbidden. -

[Help 1]

Thanx

[ankurdave]

Are you able to access the URL in the error message from a browser?

[Pallavim54]

i m not able to access this url http://repo.maven.apache.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom may be this is blocked by farewall.

and for (http://repo.maven.apache.org/maven2)

it showing this:

Browsing for this directory has been disabled.
View this directory's contents on http://search.maven.org instead.
Find out more about The Central Repository.

[Pallavim54]

yes now i m able to access the url but i m not able to run graphx example under https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala

[ankurdave]

How are you invoking it? What is the error message?

[Pallavim54]

I m getting this message when i m trying to build using maven clean install -DskipTests=true

stacktrace:

java.lang.RuntimeException: No main class detected.
at scala.sys.package$.error(package.scala:27)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37$$
anonfun$26.apply(Defaults.scala:687)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37$$
anonfun$26.apply(Defaults.scala:687)
at scala.Option.getOrElse(Option.scala:120)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37.a
pply(Defaults.scala:687)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37.a
pply(Defaults.scala:686)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:23
7)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:23
7)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestric
tions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)
error No main class detected.
[error] Total time: 1 s, completed 21 Jul, 2014 10:46:11 AM

[Pallavim54]

if i used sbt/sbt assembly then i m geeting the below error

stactrace

java.lang.RuntimeException: Nonzero exit code (1): git clone https://github.com/ScrapCodes/sbt-pom-reader.git C:\Users\pmukherjee.sbt\0.13\staging\ec3aa8f39111944cc5f
at scala.sys.package$.error(package.scala:27)
at sbt.Resolvers$.run(Resolvers.scala:134)
at sbt.Resolvers$.run(Resolvers.scala:123)
at sbt.Resolvers$$anon$2.clone(Resolvers.scala:78)
at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12$$anonfun$apply$6.apply$mcV$sp(Resolvers.scala:108)
at sbt.Resolvers$.creates(Resolvers.scala:141)
at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12.apply(Resolvers.scala:108)
at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12.apply(Resolvers.scala:108)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$3.apply(BuildLoader.scala:90)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$3.apply(BuildLoader.scala:89)
at scala.Option.map(Option.scala:145)
at sbt.BuildLoader$$anonfun$componentLoader$1.apply(BuildLoader.scala:89)
at sbt.BuildLoader$$anonfun$componentLoader$1.apply(BuildLoader.scala:85)
at sbt.MultiHandler.apply(BuildLoader.scala:16)
at sbt.BuildLoader.apply(BuildLoader.scala:142)
at sbt.Load$.loadAll(Load.scala:312)
at sbt.Load$.loadURI(Load.scala:264)
at sbt.Load$.load(Load.scala:260)
at sbt.Load$.load(Load.scala:251)
at sbt.Load$.apply(Load.scala:134)
at sbt.Load$.buildPluginDefinition(Load.scala:613)
at sbt.Load$.buildPlugins(Load.scala:579)
at sbt.Load$.plugins(Load.scala:567)
at sbt.Load$.loadUnit(Load.scala:411)
at sbt.Load$$anonfun$15$$anonfun$apply$11.apply(Load.scala:256)
at sbt.Load$$anonfun$15$$anonfun$apply$11.apply(Load.scala:256)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:93)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:92)
at sbt.BuildLoader.apply(BuildLoader.scala:143)
at sbt.Load$.loadAll(Load.scala:312)
at sbt.Load$.loadURI(Load.scala:264)
at sbt.Load$.load(Load.scala:260)
at sbt.Load$.load(Load.scala:251)
at sbt.Load$.apply(Load.scala:134)
at sbt.Load$.defaultLoad(Load.scala:37)
at sbt.BuiltinCommands$.doLoadProject(Main.scala:473)
at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:467)
at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:467)
at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
at sbt.Command$.process(Command.scala:95)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:100)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:100)
at sbt.State$$anon$1.process(State.scala:179)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:100)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:100)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.MainLoop$.next(MainLoop.scala:100)
at sbt.MainLoop$.run(MainLoop.scala:93)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:71)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:66)
at sbt.Using.apply(Using.scala:25)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:66)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:49)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:33)
at sbt.MainLoop$.runLogged(MainLoop.scala:25)
at sbt.StandardMain$.runManaged(Main.scala:57)
at sbt.xMain.run(Main.scala:29)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:129)
at xsbt.boot.Launch$.run(Launch.scala:109)
at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:36)
at xsbt.boot.Launch$.launch(Launch.scala:117)
at xsbt.boot.Launch$.apply(Launch.scala:19)
at xsbt.boot.Boot$.runImpl(Boot.scala:44)
at xsbt.boot.Boot$.main(Boot.scala:20)
at xsbt.boot.Boot.main(Boot.scala)
[error] Nonzero exit code (1): git clone https://github.com/ScrapCodes/sbt-pom-reader.git C:\Users\pmukherjee.sbt\0.13\staging\ec3aa8f39111944cc5f2\sbt-pom-reader

[Pallavim54]

can you please mention in steps that how to build the whole project and run the examples from https://github.com/apache/spark

[ankurdave]

Here's how I build and run the project. I run OS X and use SBT, not Maven.

1. Make sure Java and Git are installed:

   ~$ java -version
   java version "1.7.0_60"
   Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
   Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
   ~$ git --version
   git version 2.0.1
   

2. Clone the Spark repo:

   ~$ git clone https://github.com/apache/spark
   ~$ cd spark
   

3. Build an assembly JAR using SBT:

   ~/spark$ sbt/sbt assembly
   

4. Run tests:

   ~/spark$ sbt/sbt test
   

5. Run examples:

   ~/spark$ bin/run-example SparkPi
   

[ankurdave]

From your sbt assembly stack trace, it looks like Git is failing when running git clone https://github.com/ScrapCodes/sbt-pom-reader.git. What happens when you run that command?

[Pallavim54]

i m able to clone the repository by using git clone https://github.com/ScrapCodes/sbt-pom-reader.git

[Pallavim54]

now i m able to build an assembly JAR using SBT:
also i m able to run example using bin/run-example SparkPi.
but when i m doing ~/spark$ sbt/sbt test using SBT getting error in testcases

stacktrace:

error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
error sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 292 s, completed 21 Jul, 2014 3:03:12 PM

[Pallavim54]

Can u plz tell me the syntax of input for running this graphx test https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala .

i m using
~/spark$ bin/run-example graphx.LiveJournalPageRank soc-LiveJournal1.txt --numEPart=6

which is throwing eror ;

Exception in thread "main" java.lang.IllegalArgumentException: Invalid argument: --numEPart
at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:43)
at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:40)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.graphx.lib.Analytics$.main(Analytics.scala:40)
at org.apache.spark.examples.graphx.LiveJournalPageRank$.main(LiveJournalPageRank.scala:
at org.apache.spark.examples.graphx.LiveJournalPageRank.main(LiveJournalPageRank.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:304)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:69)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I have also tried with this run-example graphx.LiveJournalPageRank soc-LiveJournal1.txt --tol=0.001 --output=D:/output.txt --numEPart=50 --partStrategy=RandomVertexCut

but no success

[ankurdave]

That command works for me:

$ ./bin/run-example graphx.LiveJournalPageRank ~/Downloads/web-Google.txt --numEPart=8
======================================
|             PageRank               |
======================================
[...]

Is it possible you used a space instead of an equals sign, as in --numEPart 8? The error occurs if I do that.

[Pallavim54]

Geeting error with both cases:

$ ./bin/run-example graphx.LiveJournalPageRank ~/Downloads/web-Google/web-Google.txt --numEPart 8 or --numEPart=8
Exception in thread "main" java.lang.IllegalArgumentException: Invalid argument: --numEPart
at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:43)
at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:40)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.graphx.lib.Analytics$.main(Analytics.scala:40)
at org.apache.spark.examples.graphx.LiveJournalPageRank$.main(LiveJournalPageRank.scala:47)
at org.apache.spark.examples.graphx.LiveJournalPageRank.main(LiveJournalPageRank.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:304)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:69)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[Pallavim54]

can u plz let me know about this command bcoz i m not able to run this example