how to save time loading data from file - comp.soft-sys.matlab

Melding fra diskusjonen how to save time loading data from file

Rune Allnor

Vis profil

Flere alternativer 5 Aug 2008, 19:24

Nyhetsgrupper: comp.soft-sys.matlab

Fra: Rune Allnor <all...@tele.ntnu.no>

Dato: Tue, 5 Aug 2008 10:24:56 -0700 (PDT)

Lokal: Tirs 5 Aug 2008 19:24

Emne: Re: how to save time loading data from file

  Videresend | Skriv ut | Vis tråd
  | Vis original
  | Rapporter denne meldingen | Finn meldinger av denne brukeren
  

On 5 Aug, 00:10, "ggk " <ggkm...@comcast.net> wrote:

> > The only thing that will speed things *significantly*
> > is to store the data on binary format. Not too long ago
> > I sped up the loading of 90 MB of ASCII files from ~45 s
> > to ~0.2 s by changing the storage format to binary. Not
> > only were the loading some 250x faster, I also saved some
> > 20% disk space by storing the data on binary format. ...
> Wow, loading binary files goes 250x faster!? How can I
> convert my ascii file to binary? Any routines to do this
> that can be called from Matlab?

Below is a script to demonstrates the timing differences
between ASCII and binary data. Output on my screen (R2006a):

Wrote ASCII data in 2.2344 seconds
Read ASCII data in 4.1719 seconds
Wrote binary data in 0.03125 seconds
Read binary data in 0.03125 seconds

File sizes:

test.txt (ASCII) 17579 kB
test.raw (Binary) 7813 kB

Rune

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
N = 1000000;
d1=randn(N,1);
t1=cputime;
save test.txt d1 -ascii
t2=cputime-t1;
disp(['Wrote ASCII data in ',num2str(t2),' seconds'])

t3=cputime;
d2=load('test.txt','-ascii');
t4=cputime-t3;
disp(['Read ASCII data in ',num2str(t4),' seconds'])

t5=cputime;
fid=fopen('test.raw','w');
fwrite(fid,d1,'double');
fclose(fid);
t6=cputime-t5;
disp(['Wrote binary data in ',num2str(t6),' seconds'])

t7=cputime;
fid=fopen('test.raw','r');
d3=fread(fid,'double');
fclose(fid);
t8=cputime-t7;
disp(['Wrote binary data in ',num2str(t8),' seconds'])

Videresend