If you strace the dbimport process while it's working on a table
with a text/byte column you can see it repeat the following
pattern of system calls, per row inserted into the table:
20:33:42.726326 open("/tmp/dbx21339_5",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
20:33:42.734611 fstat(12, {st_mode=S_IFREG|0664, st_size=0,
...}) = 0
20:33:42.734733 mmap(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc16119d000
20:33:42.734834 write(12, "XXXXXXXXXXXXXXXXXXXXXXX "..., 304) =
304
20:33:42.734915 close(12) = 0
20:33:42.735082 munmap(0x7fc16119d000, 4096) = 0
20:33:42.735190 getuid() = 1500
20:33:42.735252 getgid() = 1500
20:33:42.735314 geteuid() = 1500
20:33:42.735386 open("/tmp/dbx21339_5", O_RDONLY) = 12
20:33:42.735475 fcntl(12, F_SETFD, FD_CLOEXEC) = 0
20:33:42.735556 sendto(10,
"\'0\'4\'0\'0\'0\'21\'0\'0\'0o\'0\'0010M=\'263TSXZ0000474044\'307\'24"...,
124, 0, NULL, 0) = 124
20:33:42.735696 read(12, "XXXXXXXXXXXXXXXXX "..., 304) = 304
20:33:42.735770 read(12, "", 0) = 0
20:33:42.735829 close(12) = 0
20:33:42.735890 sendto(10, "\'0'\'0010XXXXXXXXXXXXXXX"..., 314, 0,
NULL, 0) = 314
20:33:42.735970 recvfrom(10,
"\'0^\'0\'1\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'0\'17\'0\'20\'0\'0\'0\'1\'0\'0\'0
\'0"..., 4096, 0, NULL, NULL) = 48
I then did the following performance test using 2 different
versions of an esql/c client, and dbimport to load a single 2
column table, an integer and text field. The timings were with
the table created as raw (so no logging) and ~115 million rows.
esql/c version 1: 4 hours
esql/c version 2: 4 hours 40 minutes
dbimport : 17 hours 30 minutes
Esql/c version 1 generated all the data in memory (so didn't
read from an .unl file and located the text field using
LOCMEMORY. Esql/c version 2 read from a dbexport .unl file and
parsed the 2 fields for host variables but then located the blob
host variable also using LOCMEMORY. Dbimport technically uses
LOCUSER but it appears we've set that up to essentially behave
as if LOCFILE was used. |
