tag:blogger.com,1999:blog-494502301850787022.post4274196394807545279..comments2024-03-12T01:28:13.254-07:00Comments on Kick Start Hadoop: Enable Multiple threads in a mapper aka MultithreadedMapperBejoy KShttp://www.blogger.com/profile/03581597227249520580noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-494502301850787022.post-17427457121097596542015-06-21T23:31:03.617-07:002015-06-21T23:31:03.617-07:00Thank you so much for sharing this worthwhile to s...Thank you so much for sharing this worthwhile to spent time on. You are running a really awesome blog. Keep up this good work <a href="http://www.fita.in/big-data-hadoop-training-in-chennai/" rel="nofollow">Big Data Training Chennai</a><br />Anonymoushttps://www.blogger.com/profile/17959693011717190922noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-39226827539827803352015-04-09T04:42:52.000-07:002015-04-09T04:42:52.000-07:00Hi Ramya
Ya I visited www.hadooponlinetutor.com.I ...Hi Ramya<br />Ya I visited www.hadooponlinetutor.com.I even bought the videos also.Really the videos are very good and got at $20 only.Thanku so much Ramya.Anonymoushttps://www.blogger.com/profile/16898440001371513643noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-71996250168288837912015-03-27T04:05:27.936-07:002015-03-27T04:05:27.936-07:00It is very good information and thanks for sharing...It is very good information and thanks for sharing this.even I love to share valuable information regarding technology.Recently my friend suggested me to buy hadoop videos at www.hadooponlinetraining.com.the videos are really good and having life time acess.Anonymoushttps://www.blogger.com/profile/05011709844945155964noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-2786557269500908192015-03-26T03:12:09.008-07:002015-03-26T03:12:09.008-07:00You want big data interview questions and answers ...You want big data interview questions and answers follow this link.<br />http://kalyanhadooptraining.blogspot.in/search/label/Big%20Data%20Interview%20Questions%20and%20Answerskalyan hadoophttps://www.blogger.com/profile/02603361631961528207noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-82059121779029745272015-01-22T04:54:44.552-08:002015-01-22T04:54:44.552-08:00Hadoop is really a good booming technology in comi...Hadoop is really a good booming technology in coming days. And good path for the people who are looking for the Good Hikes and long career. We also provide<a href="http://www.cubtraining.com/hadoop-online-training/" rel="nofollow"> Hadoop online training</a>Anonymoushttps://www.blogger.com/profile/15703775778402858438noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-34257902917226622202015-01-12T06:38:08.619-08:002015-01-12T06:38:08.619-08:00Thanks for sharing this. Excel Training ,
Excel ...Thanks for sharing this. <a href="http://www.a2zexcel.com" rel="nofollow"> Excel Training </a>,<br /><a href="http://www.a2zexcel.com" rel="nofollow"> Excel Training in Delhi </a>, <a href="http://www.a2zexcel.com" rel="nofollow"> Excel Training in Gurgaon</a>Parveenhttps://www.blogger.com/profile/08145045493231389976noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-19215930274680947802015-01-10T00:06:15.354-08:002015-01-10T00:06:15.354-08:00HADOOP Online Training by hyderabadsys online Trai...HADOOP Online Training by hyderabadsys online Trainings with a fantastic and continuous staff. Our Hadoop hyderabadsys online Trainings substance outlined according to the current IT industry necessity. Apache Hadoop is having great request in the business sector, tremendous number of employment opportunities are there in the IT world. Taking into account this interest hyderabadsys online Trainings began giving Online classes on Hadoop Training through the different online traininng strategies like Gotomeeting, Webex. Hadoop internet preparing is one hot no problem that has been broadly utilized. Owing to this, there is a gigantic requirement for Hadoop web preparing Administrators. There are numerous merchants who offer Hadoop internet preparing organization preparing. Engineering development inside the most recent decade has been huge to the point that things once considered unlimited are currently regular place and capacities and employments that once obliged high abilities and broad preparing can now be performed by practically anybody.<br /><a href="%E2%80%9D" rel="nofollow"> Hadoop Online Training </a><br />Contact us:<br />India +91 9030400777 <br />Usa +1-347-606-2716<br />Email: contact@Hyderabadsys.com<br />Anonymoushttps://www.blogger.com/profile/04648427956006903637noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-34686576346627538452013-11-15T02:52:16.213-08:002013-11-15T02:52:16.213-08:00It was nice article it was very useful for me as w...It was nice article it was very useful for me as well as useful for <a href="http://123trainings.com/it-hadoop-bigdata-online-training.html" rel="nofollow">Hadoop</a> learners.thanks for providing this valuable information.<br />kumarhttps://www.blogger.com/profile/13120619498930493416noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-56271792596274534482013-11-14T22:43:18.595-08:002013-11-14T22:43:18.595-08:00I must first of all appreciate the wonderfull effo...I must first of all appreciate the wonderfull efforts put in this blog to come up with such a good platform to encourage learning of <a href="http://intellipaat.com" rel="nofollow">Hadoop training</a>Anonymoushttps://www.blogger.com/profile/15508078372084641119noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-87682573450889484332013-11-01T00:22:45.208-07:002013-11-01T00:22:45.208-07:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/11651331115978215130noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-29009456373421122022013-10-01T22:51:09.108-07:002013-10-01T22:51:09.108-07:00it's nice information and it is useful for us....<br /><br />it's nice information and it is useful for us.123trainings prvides <a href="http://hadooponlinetrainings.com/hadoop-online-training/" rel="nofollow">hadoop online training in india</a>kumarhttps://www.blogger.com/profile/13120619498930493416noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-30038275019976792192013-10-01T22:49:05.841-07:002013-10-01T22:49:05.841-07:00it's nice information and it is useful for us....it's nice information and it is useful for us.123trainings prvides <a href="http://123trainings.com/it-hadoop-bigdata-online-training.html" rel="nofollow">hadoop online training in india</a><br />to see free demo just click<a href="https://www.youtube.com/watch?v=PqbYn5LXzRw" rel="nofollow">online training hadoop demo class in hyderabad</a>kumarhttps://www.blogger.com/profile/13120619498930493416noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-87062533793759371782013-09-23T23:58:57.952-07:002013-09-23T23:58:57.952-07:00it,s a nice article and it is very useful for hado...it,s a nice article and it is very useful for hadoop learners.hadoop online trainings also provides the <a href="http://hadooponlinetrainings.com/hadoop-online-training/" rel="nofollow">hadoop online training</a>kumarhttps://www.blogger.com/profile/13120619498930493416noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-33416983760554032822013-08-28T23:05:13.166-07:002013-08-28T23:05:13.166-07:00Really good piece of knowledge, I had come back to...Really good piece of knowledge, I had come back to understand regarding your website from my friend Sumit, Hyderabad And it is very useful for who is looking for <a href="http://biginfosys.com/hadoop-online-training.html" rel="nofollow">HADOOP</a>.<br />Anonymoushttps://www.blogger.com/profile/16952375189502544500noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-35723398905149689532012-07-05T01:43:20.600-07:002012-07-05T01:43:20.600-07:00Hi Arun
This post is on a map task in a map redu...Hi Arun<br /><br />This post is on a map task in a map reduce job spawning multiple threads. From what I got your case is different, You are launching mapreduce jobs in each spawned thread in your java application. And if the number of threads are too high and if the cluster capacity is not that great then definitely it can clog. You may have to get the failed task/job logs and see what is actually the root cause. The root cause can be many in your case.Bejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-50741531442014109822012-07-04T05:50:28.907-07:002012-07-04T05:50:28.907-07:00Hi Bejoy
We have java application which spawns mul...Hi Bejoy<br />We have java application which spawns multiple threads, each thread invoke a mapreduce task. Since multiple threads invoking the mapreduse task the application is failing.<br />Could you pls let me know how to execute mapreduce tasks in multi threaded way.<br /><br />Thanks<br />Arunarunhttps://www.blogger.com/profile/05903915021642473404noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-59361573328132506032012-06-03T10:40:52.128-07:002012-06-03T10:40:52.128-07:00Hi Manish
From your requirement, looks like y...Hi Manish<br /> From your requirement, looks like you don't need MultiThreadedMapper. Use the normal map reduce , you can achieve you requirement of sharing data across mappers using distributed cache. <br /><br />Some pointers inline<br /><br />1. memory (common data structures) will be shared across multiple threads<br />>> use distributed cache<br />2. each thread will be scheduled on different core and hence should be equivalent of running multiple maps on same physical box.<br />>> muliple maps is a more cleaner approach here than going for MultiThreaded one<br />3. should be able to specify bigger split size and hence combiner efficiency should improve.<br />>> it is a trade off on parallelism vs data shuffle. more maps more parallelism.<br /><br />Regards<br />BejoyBejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-88145565849122066002012-06-02T10:34:12.184-07:002012-06-02T10:34:12.184-07:00In our case, maps are memory and cpu bound. we are...In our case, maps are memory and cpu bound. we are also planning to use multithreaded mapper to achieve efficiencies in many aspects <br />1. memory (common data structures) will be shared across multiple threads<br />2. each thread will be scheduled on different core and hence should be equivalent of running multiple maps on same physical box.<br />3. should be able to specify bigger split size and hence combiner efficiency should improve.<br /><br />I have following queries<br />1. what are dis-advantages of multithreaded mapper beyond mentioned in notes section?<br />2. what could be other options to increase efficiency if maps are cpu and memory bound, primarily cpu bound.<br /><br />Manish Jain<br />Gauvus, GuavusManish Jainhttps://www.blogger.com/profile/10234758630808605542noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-52377577677805088692012-04-15T10:12:16.651-07:002012-04-15T10:12:16.651-07:00Hey,
If you are using NLineInputFormat with yo...Hey,<br /> If you are using NLineInputFormat with your spec it is simple, 5 lines in a mapper task instance. 20 Lines then 4 map tasks. When jvm reuse is -1, all the map tasks on the same node/task tracker will be using the same jvm instance. To be noted, you can never guarantee that all the mappers would be on the same node, it'd be dependent on factors like the slots available, scheduling used etc.<br /><br />Regards<br />BejoyBejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-57480474044354147402012-03-29T12:03:07.811-07:002012-03-29T12:03:07.811-07:00Finds useful.
Few questions
If I am setting mapre...Finds useful. <br />Few questions<br />If I am setting mapred.job.reuse.jvm.num.<br />tasks= -1 and my NLineInput format has value 5 and total lines is 20, how it internally executed?Unknownhttps://www.blogger.com/profile/18385984839202329050noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-68493237228018373722012-03-29T12:01:08.795-07:002012-03-29T12:01:08.795-07:00This comment has been removed by the author.Unknownhttps://www.blogger.com/profile/18385984839202329050noreply@blogger.com