tag:blogger.com,1999:blog-494502301850787022.post3038429453488709899..comments2024-03-12T01:28:13.254-07:00Comments on Kick Start Hadoop: Enable Sorted Bucketing in HiveBejoy KShttp://www.blogger.com/profile/03581597227249520580noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-494502301850787022.post-3979299577436855242015-02-16T01:31:30.614-08:002015-02-16T01:31:30.614-08:00Bejoy,
Was really useful information. I have a qu...Bejoy,<br /><br />Was really useful information. I have a question which I am posing as a use-case.<br /><br />Say that there exists a bunch of ids that come from an external source. And there is the HDFS that has buckets hashed by id and ordered by id. There is a need to delete the records corresponding to the external ids.<br /><br />1. Can I delete the records from a specific bucket ? <br />2. How can I make this process optimal ?<br />3. If deletions can take place at the bucket level, what happens if a few records are deleted ? Does it create a new bucket with the remaining data or are the deleted rows marked as null ?<br />4. What happens if all the rows in a bucket are deleted ? <br />The Beginner's Mindhttps://www.blogger.com/profile/07686846093597540987noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-5276954979734673262013-06-16T10:10:23.978-07:002013-06-16T10:10:23.978-07:00The following is directly from the hive wiki:
>...The following is directly from the hive wiki:<br /><br />>> Cluster By is a short-cut for both Distribute By and Sort By. https://cwiki.apache.org/Hive/languagemanual-sortby.htmlStephen Boeschhttps://www.blogger.com/profile/08054737943803985586noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-49435182756964628452013-06-16T09:44:34.539-07:002013-06-16T09:44:34.539-07:00Ho Bejoy, would you be willing to explain why the ...Ho Bejoy, would you be willing to explain why the "cluster by" - which is a synonym for (distribute by && sort by) would then require (apparently redundant) "sort by" as well?Stephen Boeschhttps://www.blogger.com/profile/08054737943803985586noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-81432026861751013932013-03-26T07:44:56.570-07:002013-03-26T07:44:56.570-07:00Hi I am not able execute any of the above queries ...Hi I am not able execute any of the above queries even loaded with datatiruhttps://www.blogger.com/profile/05148747628433396198noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-23521427368450290392013-03-25T05:58:52.641-07:002013-03-25T05:58:52.641-07:00To know the partitioning and bucketing information...To know the partitioning and bucketing information regarding an existing table use<br />DESCRIBE FORMATTED/EXTENDED Bejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-23194862499398049202013-02-05T23:50:38.171-08:002013-02-05T23:50:38.171-08:00how would we know how many partitioned bucketed in...how would we know how many partitioned bucketed in table?<br /><br />Is there any query it show before we use (bucket 3 out of n)<br />Please help me any body on this.NarendraMuthumhttps://www.blogger.com/profile/15075050608481313407noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-68049362125447501142012-09-10T10:18:50.870-07:002012-09-10T10:18:50.870-07:00Hi Jeet
10 tables on a join has limited scope of ...Hi Jeet<br /><br />10 tables on a join has limited scope of optimization. If you can break down the query then you can utilize the various optimization techniques and apply them individually. I can comment on it in a more detailed manner only if I know your use case, queries, data volume involved for each table and your cluster statistics. <br /><br />What is Bucketed Map Join?<br />If two tables are bucked o the same key/keys on which the join is done and if one of the tables are medium sized then you can benefit from bucketed map join. When an input split from large table is processed by a mapper the corresponding bucket from smaller table can be loaded in memory and achieve a map side join and there by eliminating the reduce phase. This speeds up the join operation to a greater extent.Bejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-63534970799675500552012-08-25T11:52:25.759-07:002012-08-25T11:52:25.759-07:00Can you Explain, how bucketing can optimize join?
...Can you Explain, how bucketing can optimize join?<br />Ex : i have 10 tables in hive.<br />I want to create a new table which will be a join result of all 10 tables.<br />Another question is can i do the join of 10 table in a single query and store the output in 1 table? and what i can think about to optimize the join query in hive(especially when it is a 10+ table join)?I need serious help on this.<br />Please explain it by the help of a code.jeethttps://www.blogger.com/profile/13300088306027124935noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-56449511324330958062012-06-03T10:32:14.495-07:002012-06-03T10:32:14.495-07:00Hi Tiru,
Bucking is basically used for sampling a...Hi Tiru,<br /><br />Bucking is basically used for sampling and not for queries that point to certain data group. Say you want to calculate the average of age(one column in table) and you not looking at exact average but an approximate one. In those cases bucketing is used . Also bucketing is used to optimize joins as well. <br /><br />Also when I mentioned '2 of n' buckets means, choose 2 buckets out of n. If I mention 'n on n' then whole data would be used as input.<br /><br />Regards<br />BejoyBejoy KShttps://www.blogger.com/profile/03581597227249520580noreply@blogger.comtag:blogger.com,1999:blog-494502301850787022.post-40493679576952657252012-05-29T01:57:36.323-07:002012-05-29T01:57:36.323-07:00hi
very good post.
i have one question.
How do we...hi <br />very good post.<br />i have one question.<br />How do we know that required data is resided in 2 bucket?<br /><br />thanks <br />tirutiruhttps://www.blogger.com/profile/05148747628433396198noreply@blogger.com