A New Approach to Extraction Transformation Loading Using Pipelining
Keywords:
Data Warehouse, Data Mart, Pipelining, Nested PipeliningAbstract
Companies have lots of precious data lying around throughout their networks that needs to be moved from one place to another such as from one business centre to another or to a data warehouse for analysis. The problem is that the data lies in different sorts of heterogeneous systems, and therefore in all sorts of formats. To accumulate data at one place and to make strategic decisions we need a data warehouse system. This consists of extract, transform and load (ETL) software, which extract data from various resources, transform it into new data formats according to required information needs, and then load it into desired data structure(s). Such softwares take enormous time for the purpose. To deal with the problem of time taken by ETL process, in this paper we are presenting a new technique of ETL using pipelining. We had divided ETL process into various segments which worked simultaneously using pipelining.
References
• Floria11a l!'.,po.l'ito, " A Compartlfi,,,, a11al1•.,is 111 1111.,h 1 . I) . . "IF.'·' . ~ II( I 1,,,. flrt/11111~ ,•n.mm trc,•s, .i:.E 7 r11111'tu·,,· 1 • • I /I 11/J iit1ffl'l'II a11a/ysis mu/ paffem mc1tchi11g rol 19, f91J7. • P. Vassiliadi.1-, C. Quix, Y. Vassiliuu. M. lt1rk,· I>,, 1m1•t1 rdw11.1·11 P ron•.1·s11,arwgeme111. l11Jon11w1<11; Sv.1·t11ms. 1•01. 2ti. 110.J, pp. 205-236. Jwu, 2001 • '/: Stiilrr. R. Miil/er, E. Rt1hm. An lntt•gmtil-r mi.I Uniform Modi'! for Mett1dma Mmwgeme111 in 1)11111 Wal'l'lro11.1·ing E11viro11mt•nts. In Proc11edi11gs ,if tire /nti•mati/1/llll Workshop on Desig11 and Managm11·n1 of Data Warelro11.1·n ( DMDW'99), pp. 12. I - 12.16. Heidelb,1 rg. German_,·, 1999.
• !nmo11. W.H. , R11ildi11g tire Data Warelu111.ve. John Wiley. /992
• M. Jarke, M. Len:aini. Y. Vassiliou. P. Vassiliadis ( ed.1·. ). Fwrdamenwls of Datt1 Wart1lro11ses. 2nd Edition. Springt•r-Verlag, Germany. 2003
• w11•11•. comp11ten 11orld.com.
• lrffp.l/en. wikipedia.org/wiki/Pipeline
• Kimball, Ra/pl,; Joe Caserta (200-1 ). Thi• Datu Ware/,cmse ETL Toolkit. Wiley. ISBN 0-76-15-6757-8. • Kimball, Ralph: Margy Ross (2002). Thi• Datu
Warl'lw11se Toolkit: Tire Compfrt1' Guidt' to Dimensional Modeling ( 2nd edition t'd. ). Wiley. pp. 358-362. ISBN 0-471-20024-7.
• Kimball, Ralph: et al. ( /998). The Daw 'arelwus<' Lifecycll' Toolkit. Wiley. ISBN 0-471-25547-5 • lrftp://en.wikipedia.org/wiki/Etl#Transfomr • lrffp:/lwww.mo11fto11.com/olap/olap.glMsa0•. /r1111I • J.A. Blakelev. N. Cob11m. a11d P. -A. Lilrscm. Up-dating derived retaiions: electing irrelevant and a11tonomil/lsly compwable updmes. ACM Transactions on Datt1bcisf Systems, 1-1(3):369/400, September /989
• lwp:llf'fl. wikipedia. orglwiki/Etl#Trcmsfonn • lrffp:llwww. mo11lto11. ccmi/olap/olap.glossary.lrtml
Downloads
Published
Issue
Section
License
Copyright (c) 2010 Trinity Journal of Management, IT & Media (TJMITM)
This work is licensed under a Creative Commons Attribution 4.0 International License.
