Wednesday 18 March 2015

Apache Spark and Hadoop Integration with example

Step 1 : Install hadoop in your machine  ( 1x or 2x) and also you need to set java path and scala path in .bashrc ( for setting path refer this post Spark installation )

Step 2: Check all hadoop daemons are up running.



Step 3: Write some data in your hdfs (here my file name in hdfs is word)

Step 4: Download apache spark for hadoop 1x or 2x based on you hadoop installed version in step 1.



Step 5: Untar the spark hadoop file.

Step 6: Start the spark hadoop shell.



Step 7: Type the following command.once spark shell started



Step 8: See the Out put In terminal.



Step 9: Check the Namenode UI (localhost:50070)







Step 10: Check the spark UI (localhost:4040) for monitoring the job


Posted by at

79 comments:

  1. Unknown20 July 2015 at 00:28

    There are lots of information about hadoop have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get to the next level in big data. Thanks for sharing this.

    Hadoop Training in Chennai
    Big Data Training in Chennai
    Hadoop training velachery

    ReplyDelete
  2. Unknown25 March 2016 at 11:21


    Nice information about Hadoop.
    The best place to learn hadoop is steinmetzils
    100% Job assurence is provided by them.
    visit: http://www.steinmetzils.com/

    ReplyDelete
  3. Unknown25 March 2016 at 11:21


    Nice information about Hadoop.
    The best place to learn hadoop is steinmetzils
    100% Job assurence is provided by them.
    visit: http://www.steinmetzils.com/

    ReplyDelete
  4. Unknown12 June 2016 at 05:06

    Great Info about spark. I have written some more posts about spark. see here : https://lets-do-something-big.blogspot.in/2016/06/custom-udf-in-apache-spark.html

    ReplyDelete
  5. Unknown12 June 2016 at 05:06

    This comment has been removed by the author.

    ReplyDelete
  6. Unknown12 June 2016 at 05:07

    This comment has been removed by the author.

    ReplyDelete
  7. Unknown12 June 2016 at 05:07

    This comment has been removed by the author.

    ReplyDelete
  8. Unknown21 March 2017 at 00:21

    Thanks for sharing your post.You con also visit on
    Best Hadoop Training Institute in Gurgaon

    ReplyDelete
  9. Sujitkumar3 May 2017 at 03:29

    Thanks for sharing Valuable information. Greatful Info about hadoop. Really helpful. Keep sharing........... If it possible share some more tutorials.........

    ReplyDelete
  10. srikanth20 May 2017 at 03:27

    Apache Spark today remains the most active open source project in Big Data with over 1000 contributors. Spark offers over 80 high-level operators that make it easy to build parallel apps
    apache spark developer training

    ReplyDelete
  11. Unknown8 August 2017 at 22:45

    Great information. Learn Hadoop from SV Soft Solutions. They have experts to teach Hadoop with real time examples. You can check Hadoop Curriculum at their website. Our trainers feed you with the best training materials , hadoop online training &online sessions and a real time scenario.

    ReplyDelete
    Replies
    1. Unknown8 August 2017 at 22:46

      The trainer contact number is +1-845-915-8712, +91-9642373173

      Delete
  12. santhosh k7 September 2017 at 05:13

    The Example about the apache spark and hadoop integeration was very much useful my sincere thanks for sharing this post
    Hadoop Training in Bangalore

    ReplyDelete
  13. Unknown2 November 2017 at 03:28

    I definitely appreciate your blog. Excellent work!
    startups

    ReplyDelete
  14. rose2 November 2017 at 03:42

    It is really a great work and the way in which you are sharing the knowledge is excellent.Thanks for your informative article
    selenium training in bangalore|
    python training in bangalore|

    ReplyDelete
  15. Unknown2 November 2017 at 22:49

    Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.
    digital training in chennai

    ReplyDelete
  16. Unknown15 November 2017 at 04:52

    your giving the nice information about the big data, Hadoop this is very essential for the peoples. In future also post this type of information.
    http://www.kellytechno.com/Bangalore/Course/Hadoop-Training

    ReplyDelete
  17. Unknown15 November 2017 at 04:54

    your giving the nice information about the big data, Hadoop this is very essential for the peoples. In future also post this type of information.
    http://www.kellytechno.com/Bangalore/Course/Hadoop-Training

    ReplyDelete
  18. Unknown24 November 2017 at 02:44

    Hai,
    It's very nice blog
    Thank you for giving valuable information on Hadoop
    I'm expecting much more from you...

    ReplyDelete
  19. Thanuja Sri3 December 2017 at 09:46

    Nice post about hadoop.are you looking for hadoop online training .

    ReplyDelete
  20. Unknown11 December 2017 at 04:57

    Thank you for posting the nice Information.
    Hadoop Training In Bangalore

    ReplyDelete
  21. soukya reddy15 December 2017 at 04:19

    Nice. Thank you for sharing big data hadoop online training bangalore

    ReplyDelete
  22. Aptron15 December 2017 at 05:04

    Best Topic in the Big data training... Nice & Appreciable Blog

    Aptron India

    ReplyDelete
  23. vamsi17 December 2017 at 22:43

    Hi,

    I have read your Hadoop blog it's very attractive and impressive. I like it your blog....

    Thank you
    priya

    ReplyDelete
  24. Robertreynold20 December 2017 at 04:17

    Hi,
    Thanks for sharing such a nice informative post
    Big Data Analytics Training In Hyderabad

    ReplyDelete
  25. Robertreynold20 December 2017 at 04:19

    Hi,
    Thanks for sharing such a nice article
    Big Data Analytics Training In Hyderabad

    ReplyDelete
  26. Kumar Ranjan23 December 2017 at 03:57

    Nice post.Very useful and informative one.Keep sharing
    Big Data Training in Gurgaon

    ReplyDelete
  27. Unknown26 December 2017 at 08:19

    Thanks for sharing this blog post. Its very informative blog. big data hadoop institution in chennai

    ReplyDelete
  28. Mahesh28 December 2017 at 04:32

    Thanks for sharing this blog. This very important and informative blog for Hadoop etc
    I really loved reading this. Keep it up.!!
    Learned a lot of new things from your post! Good creation and HATS OFF to the creativity of your mind.
    Very interesting and useful blog!
    Best Hadoop training in gurgaon

    ReplyDelete
  29. Mahesh28 December 2017 at 04:32

    Thanks for sharing this blog. This very important and informative blog for Hadoop etc
    I really loved reading this. Keep it up.!!
    Learned a lot of new things from your post! Good creation and HATS OFF to the creativity of your mind.
    Very interesting and useful blog!
    Best Hadoop training in gurgaon

    ReplyDelete
  30. Prwatech3 January 2018 at 21:24

    Nice Post. Thank you for sharing big data hadoop online training bangalore

    ReplyDelete
  31. David10 January 2018 at 05:04

    Thanks for sharing this article about apache spark and hadoop...
    Hadoop Training in Hyderabad

    ReplyDelete
  32. Shailendra11 January 2018 at 03:24

    Hello....your blog is awesome and thanks for Posting. i read your content thats very interesting so please keep it up.
    Hadoop Training Classes in Gurgaon

    ReplyDelete
  33. Unknown17 January 2018 at 04:21

    Thanks For Sharing the Nice Information About the Top Bigdata.I have read a lot of blogs about the Bigdata. But your Blog has the cool information.The information that you shared is very helpful for me.
    http://www.kellytechno.com/Hyderabad/Course/Hadoop-Training

    ReplyDelete
  34. Unknown18 January 2018 at 00:08

    Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this. Big Data Training Institute in Chennai

    ReplyDelete
  35. Mahesh23 January 2018 at 03:20

    You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Hadoop training in Gurgaon



    ReplyDelete
  36. Mahesh23 January 2018 at 03:21

    You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Hadoop training in Gurgaon



    ReplyDelete
  37. raju v23 January 2018 at 23:15



    Awesome,
    Thank you so much for sharing such an awesome blog...
    Training and job placement services
    Data Science Training placements services
    Advanced analytics certification

    ReplyDelete
  38. Unknown25 January 2018 at 23:01

    Nice blog. Really helpful for learning Big Data Training and keep update on some more tutorials….. I liked your blog.

    ReplyDelete
  39. Mahesh27 January 2018 at 03:35

    You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Hadoop Training

    ReplyDelete
  40. Mahesh27 January 2018 at 03:36

    You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Hadoop Training

    ReplyDelete
  41. logistic-solutions7 February 2018 at 02:10


    Thank you for your post. This is excellent information. It is amazing and

    wonderful to visit your site.
    big data and analytic solutions

    ReplyDelete
  42. logistic-solutions15 February 2018 at 21:07

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    big data & analytics services in NA

    ReplyDelete
  43. raju v18 February 2018 at 23:58

    Awesome,
    Thank you so much for sharing such an awesome blog...
    sas certified big data professional
    self paced E learning program

    ReplyDelete
  44. poshesolutionsinfo22 February 2018 at 23:00

    easily understandable information with above example for all people:)

    ReplyDelete
  45. raju v25 February 2018 at 23:15

    Awesome,
    Thank you so much for sharing such an awesome blog...
    data science training services
    data science training placements services

    ReplyDelete
  46. BrnInfotech2 March 2018 at 02:48

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    Iphone training course

    ReplyDelete
  47. BrnInfotech2 March 2018 at 02:49

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    Iphone training course

    ReplyDelete
  48. BrnInfotech2 March 2018 at 02:49

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    Iphone training course

    ReplyDelete
  49. technogeeks115 March 2018 at 03:25

    This comment has been removed by the author.

    ReplyDelete
  50. technogeeks115 March 2018 at 03:29

    Nice information about Hadoop. Thank you For Sharing. For More About hadoop admin training in pune Visit: https://technogeekscs.com/

    ReplyDelete
  51. rambo989814 March 2018 at 04:50

    Its nice information . apart from the text graphical information is more effective .

    ReplyDelete
  52. logistic-solutions19 March 2018 at 02:50

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    sap supplier diversity management services
    Big Data Analytics Consulting for Hadoop

    ReplyDelete
  53. Digitoliens19 March 2018 at 18:39

    nice information sir, if you want to see best hadoop training institute in hyderabad visit us @ vr it solutions

    ReplyDelete
  54. BrnInfotech19 March 2018 at 22:21


    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    Best iphone training in hyderabad

    ReplyDelete
  55. Kritika21 March 2018 at 00:15

    Great post. Thank you for sharing such useful information. Please keep sharing
    Big Data Training in Delhi

    ReplyDelete
  56. Nbits21 March 2018 at 22:53

    Good post. Please keep share like this
    hadoop training in hyderabad

    ReplyDelete
  57. Nbits21 March 2018 at 22:54

    good post thanks for sharing please keeep share like this
    hadoop training in hyderabad

    ReplyDelete
  58. raju v22 March 2018 at 02:07

    Awesome,
    Thank you so much for sharing such an awesome blog...
    sas data training placement support
    best sas training center

    ReplyDelete
  59. joe connors2 April 2018 at 01:51

    Hi, Thanks for sharing the information. This information will really help us a lot.
    Hadoop Training in Hyderabad

    ReplyDelete
  60. joe connors2 April 2018 at 01:53

    Hi, Thanks for sharing the information. This information will really help us a lot.

    Hadoop Training in Hyderabad

    ReplyDelete
  61. Unknown6 April 2018 at 00:23

    thanks for the post the bigdata installation was given in simplified manner and very understandable ,good job
    Hadoop Training in Velachery | Hadoop Training .

    ReplyDelete
  62. Unknown6 April 2018 at 00:23

    thanks for the post the bigdata installation was given in simplified manner and very understandable ,good job
    Hadoop Training in Velachery | Hadoop Training .

    ReplyDelete
  63. Unknown10 April 2018 at 04:39

    Privileged to read this informative blog on Hadoop.Commendable efforts to put on research the hadoop. Please enlighten us with regular updates on hadoop. Friends if you're keen to learn more about AI you can watch this amazing tutorial on the same.
    https://www.youtube.com/watch?v=1jMR4cHBwZE

    ReplyDelete
  64. Unknown15 April 2018 at 22:59

    Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.

    big data training in chennai

    ReplyDelete
  65. Unknown15 April 2018 at 23:01

    Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.

    big data training in chennai

    ReplyDelete
  66. puspraj kumar18 April 2018 at 00:12

    big data training institutes in Bangalore gives the students in-depth knowledge regarding this happening subject so that they can take their training and apply it where needed. Some highly qualified people take classes at the spark training institutes in Bangalore so that the students can learn the subject thoroughly complete with practical applications. http://www.npntraining.com/courses/apache-spark-scala-training.php

    ReplyDelete
  67. Techtoolsinnovation27 April 2018 at 04:33

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    sas analytics training

    ReplyDelete
  68. SLA5 May 2018 at 02:27

    This comment has been removed by the author.

    ReplyDelete
  69. SLA5 May 2018 at 02:30

    A great example you have cleared now after long days i have been looking.

    Big data training in chennai

    ReplyDelete
  70. ohhanibaek8 May 2018 at 04:23

    Hi,
    Good Information,Thanks for sharing the topic on apache spark and hadoop integration.it is very useful to all spark and hadoop developers.nice blog
    Spark Training in Hyderabad

    ReplyDelete
  71. ohhanibaek8 May 2018 at 04:27

    Good information posting.
    thank you
    Spark Training in Hyderabad

    ReplyDelete
  72. marry14 May 2018 at 00:16

    Thanks for sharing this Informative content.
    Big Data Training in Delhi

    ReplyDelete
  73. logistic-solutions17 May 2018 at 00:28

    Thanks for posting the useful information to my vision. This is excellent information
    Awesome,
    big data analytics consulting companies
    big data and analytic solutions

    ReplyDelete
  74. logistic-solutions23 May 2018 at 23:25

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    sap business intelligence services

    ReplyDelete
  75. Techtoolsinnovation24 May 2018 at 00:33

    Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    with Placement assistance

    ReplyDelete
  76. Unknown27 May 2018 at 03:39

    Thanks for sharing this great information- https://www.madridsoftwaretrainings.com/hadoop.php

    ReplyDelete
  77. Unknown28 May 2018 at 21:36

    Such a nice post, I like your Post.This is too much informational. Typistpool Analytics can play a main role in resolving some realistic hard business difficulties. Typistpool provide best quality data analytics consulting services including data tabulation, data collation, data visualization, data discovery.

    ReplyDelete
  78. Unknown30 May 2018 at 04:45

    This Big Data Admin Hadoop training course is a comprehensive training designed by industry experts considering current industry job requirements to provide in-depth learning on Big Data and Hadoop Admin Modules. Cloud Lab access will be provided for a month.

    To know for more result please visit once - http://www.knowledgeera.co.in/training/Big-Data-Hadoop/11

    ReplyDelete

Subscribe to: Post Comments (Atom)