Already a member?
Sign in
Welcome! This is a website that everyone can build together. It's easy!
Location: Data Import and Export
Discussion: Data Pump.....and Compression?
Watch
 psmith@epx.com |
Data Pump.....and Compression?
Dec 17 2007, 3:58 PM EST A common technique used with exp/imp was compressing through a Pipe. Apparently this cannot be done with data pump "the ... dumpfile ... is no longer processed in a sequential manner" (Note:463336.1). Certainly understandable up to the point of output to the named DUMPFILE...but beyond that point isn't this a fundamental violation of the accepted handling of Files in an UNIX environment? The specified DUMPFILE doesn't comply with what I've always understood to be the basic rules of Files in UNIX. Perhaps it might make some sense to allow the DUMPFILE one additional level of abstraction by using a BLOB or External Table for non-sequential assembly, the attaching this to a Standard File output which would adhere to the traditional conventions...once again allowing use of Pipes! Just an idea..... 1 out of 14 found this valuable. Do you? |
|
Moles Moles |
RE: Data Pump.....and Compression?
Jan 21 2008, 3:21 PM EST On the fly compression using gzip or similar can easily reduce dump space requirements by a factor of 5 or more. For large databases that is very significant indeed. Furthermore, if the processing capacity is available, the resultant reduction in I/O required to process a dump will substantially speed up the whole process. Since the expdp utility effectively prevents use of pipes because, I quote from the manual "If there are preexisting files that match the resulting filenames, an error is generated. The existing dump files will not be overwritten.' one would appear to be screwed even before considering how to handle multiple o/p files. Trying to circumvent this by using the DBMS_DATAPUMP API doesn't work either. I understand that the compression issue has been addressed in11g. Anyone want to confirm or deny that based on experience? 1 out of 12 found this valuable. Do you? |
 rucknrun |
RE: Data Pump.....and Compression?
Jan 25 2008, 10:16 AM EST We gzip up our dump files after they are created. I have set up our nightly exports to gzip up the files. If you script your backups using a shell script you can easily make the filenames unique. We attach the time to the end of the filename. 1 out of 12 found this valuable. Do you? |
|
Moles Moles |
RE: Data Pump.....and Compression?
Jan 25 2008, 11:21 AM EST Having to compress after the fact is very wasteful. Even if you don't care about the disk space - and with very large databases that is unlikely to be the case - it wastes as lot of resources. Somewhere between two and three times as much non-cached sequential I/O will be required to process the dump files. Since I/O is the slowest part of the system that means both imports and exports take far longer than would be the case if Oracle Corp. had built data pump better. In a Standard Edition environment (which does not provide for parallel dump file activity), it's quite likely that the overall performance of the standard/old export utility when used with compression and named pipes will be comparable to data pump. Thus the only compelling reason for using data pump is for those databases using data types that export does not support. A sad state of affairs. 2 out of 12 found this valuable. Do you? |
|
rucknrun |
RE: Data Pump.....and Compression?
Jan 25 2008, 1:38 PM EST Well, gzip is probably more efficient at compression then the Oracle process would be. I don't think compressing the dumps after the fact is that big a deal. 2 out of 17 found this valuable. Do you? |
