Oracle provides docker images for its database, is there a way to reduce the size of the docker image down from the current 5.06GB? That size is obtained building with the defaults. Tested on 12.1.0.2-se2.
Related
I'm trying to import a data dump from S3 bucket to Oracle RDS instance following this guide.
However my dump is around 10GB in size & when I try to download it from S3 bucket to data_pump_directory using below procedure, it gives below error.
SELECT rdsadmin.rdsadmin_s3_tasks.download_from_s3(
p_bucket_name => '<bucket_name>',
p_directory_name => 'DATA_PUMP_DIR')
AS TASK_ID FROM DUAL;
Error: Insufficient disk space to download, at least 10724474880 bytes required
My current data_pump_directory size is 4GB.
Want to know how to increase it. I tried increasing the storage of RDS & upgrading the DB class, but no luck.
TIA
You can take the easy route and increase the storage size.
AWS->Documentation->Amazon Relational Database Service (RDS)->User Guide->
Increasing DB instance storage capacity
But there is another option.
.Make export data expdp with the option filesize = 2GB
Copy file one by one and import, delete dump file and repeat several
times.
It seems to me your RDS Oracle instance use more storage than expected, by default all tablespaces created should have auto-extend = true, the DATA_DUMP_DIR simply use the same storage as the data files, which means if you increase the storage size of the RDS instance should fix the issue.
Please have a look at this: https://aws.amazon.com/premiumsupport/knowledge-center/rds-oracle-storage-optimization/, to check how much storage is allocated for each tablespace and how to clean up the DATA_DUMP_DIR if needed.
AWS RDS also got the feature of "storage autoscaling", you can try to enable that as well by following this: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling
I have a requirement where I need to fetch blob files within subdirectory of a blob container.
Problem is I will not know the container name as containers are dynamically created.
I do not want to fetch list of containers, iterate through them and then get list of blobs. Just trying to avoid loops since there could be many containers.
Is there any simple way of getting all blobs from all containers without knowing container name?
Regards,
Madhu
Simple answer to your question is that it's not possible to get a list of blobs without knowing the container name as blobs belong to a container. You would need to know the name of the blob container to list the blobs in that container.
My organization is thinking about offloading the unstructured data like Text , images etc saved as part of Tables in Oracle Database , into Hadoop. The size of the DB is around 10 TB and growing. The size of the CLOB/BLOB columns is around 3 TB.Right now these columns are queried for certain kind of reports through a web application. They are also written into but not very frequently.
What kind of approach we can take to achieve proper offloading of data and ensuring that the offloaded data is available for read through existing web application.
You can get part of the answer in oracle blog (link).
If data needs to be pulled in HDFS environment via sqoop, then you must first read the following from sqoop documentation.
Sqoop handles large objects (BLOB and CLOB columns) in particular ways. If this data is truly large, then these columns should not be fully materialized in memory for manipulation, as most columns are. Instead, their data is handled in a streaming fashion. Large objects can be stored inline with the rest of the data, in which case they are fully materialized in memory on every access, or they can be stored in a secondary storage file linked to the primary data storage. By default, large objects less than 16 MB in size are stored inline with the rest of the data. At a larger size, they are stored in files in the _lobs subdirectory of the import target directory. These files are stored in a separate format optimized for large record storage, which can accomodate records of up to 2^63 bytes each. The size at which lobs spill into separate files is controlled by the --inline-lob-limit argument, which takes a parameter specifying the largest lob size to keep inline, in bytes. If you set the inline LOB limit to 0, all large objects will be placed in external storage.
Reading via web application is possible if you are using MPP query engine like Impala and it works pretty well and it is production ready technology. We heavily use complex Impala queries to render content for SpringBoot application. Since Impala runs everything in memory, there is a chance of slowness or failure if it is multi-tenant Cloudera cluster. For smaller user groups (1000-2000 user base) it works perfectly fine.
Do let me know if you need more input.
Recommendation will be
Use Cloudera distribution (read here)
Give enough memory for Impala Deamons
Make sure you YARN is configured correctly for schedule (fair share or priority share) based ETL load vs Web Application Load
If required keep the Impala Daemons away from YARN
Define memory quota for Impala Memory so it allows concurrent queries
Flatten your queries so Impala runs faster without joins and shuffles.
If you are reading just a few columns, store in Parquet, it works very fast.
We have an unsupported Oracle application running 9i as the database server and 10g on the application server.
As part of the application we are able to load and save photos (jpeg) within the database. We have started using the photos for different purposes and I have noticed that there is a huge difference in file size where comparing the preload file with the image when exported. Can anyone explain why this is happening.
As an example when we load a 10mb file, the photo is only just over 1mb when extracted. There is on average an 84% decrease in the exported file size over the original photo. The image dimensions are still the same size as is the dpi. Any ideas why the exported images are significantly smaller then the images loaded to the database.
Thanks,
I'm working with a database-driven application that allows users to upload images which are then zipped and embedded into a database in a varbinary(max) format. I am now trying to get that image to display within an SSRS report (using BI 2005).
How can I convert the file data (which is 65,438 characters long when zipped and 65,535 characters when not zipped) into a normal varbinary format that I can then display in SSRS?
Many thanks in advance!
You'll have to embed a reference to a dll in your project and use a function to decompress the data within SSRS, see for example SharpZipLib. Consider storing the data uncompressed if possible, as the CPU / space trade off is unlikely to be in your favour here, as impage data is likely to have a poor compression ratio (it is usually already compressed).