---------------------------------------------------------------
[ENHANCED] VFP 9.0 FIX - COMPARE LARGE STRINGS
January 2024
---------------------------------------------------------------
CCB
1. BUG:
If we enable comparing large strings in Visual FoxPro Advanced,
it can compare 2 strings when the length of each string < 8 MB.
It is the startup default for Visual FoxPro Advanced.
If we disable comparing large strings in Visual FoxPro Advanced,
it can compare 2 strings when the length of each string < 300 KB.
The bug only occurs when SET("COLLATE") != "MACHINE", it does not occur when SET("COLLATE") == "MACHINE".
The bug only occurs in the following functions and operators:
BETWEEN() function, INLIST() function,
< operator, <= operator, != operator, = operator, > operator, >= operator, == operator.
2. CAUSE:
When SET("COLLATE") != "MACHINE", to compare strings in Visual FoxPro 9.0 and earlier versions,
it will use stack space to save some temporary data.
Usually the default stack size is about 1 MB (in fact, the stack size is about 920 KB in Visual FoxPro),
it can compare 2 strings when the length of each string < 300 KB.
When SET("COLLATE") != "MACHINE", to compare strings in Visual FoxPro Advanced,
it will use available memory to save some temporary data,
it can compare 2 strings when the length of each string < 8 MB.
If the length of each string <= 64 KB,
it will use stack space to save some temporary data,
it is the same as Visual FoxPro 9.0 and earlier versions.
Usually, to compare strings x1 and x2,
it needs more memory to save the sort key data:
LEN(x1)*4 ;; the maximum length of the generated sort key (LCMAP_SORTKEY)
LEN(x1)*54 ;; internal using in the LCMapString() function
so it needs memory LEN(x1)*4 + LEN(x2)*4 + MAX(LEN(x1)*54 , LEN(x2)*54).
For example, when SET("COLLATE") == "PINYIN",
x1=REPLICATE("a",8000000)
x2=REPLICATE("b",8000000)
it needs memory 8*4 + 8*4 + MAX(8*54 , 8*54) = 496 MB,
for the expression x2 > x1,
usually it will run fine.
For example, when SET("COLLATE") == "PINYIN",
x1=REPLICATE("a",40000000)
x2=REPLICATE("b",40000000)
it needs memory 40*4 + 40*4 + MAX(40*54 , 40*54) = 2480 MB > 2 GB,
for the expression x2 > x1,
it will cause the error "There is not enough memory to complete this operation (Error 43)".
When SET("COLLATE") != "MACHINE", to compare strings in SQL statements in Visual FoxPro 9.0 and earlier versions,
usually it can only compare 2 strings when the length of the generated sort key (LCMAP_SORTKEY) < 240.
For example, when SET("COLLATE") == "PINYIN",
usually it can only compare 2 strings and return the correct result when the length of each string < 120.
If the length of the generated sort key (LCMAP_SORTKEY) >= 480,
it will cause the error "Expression evaluator failed (Error 67)" or other errors.
If the length of the generated sort key (LCMAP_SORTKEY) >= 240,
it will return the incorrect result.
When SET("COLLATE") != "MACHINE", to compare strings in SQL statements in Visual FoxPro Advanced,
usually it can only compare 2 strings when the length of the generated sort key (LCMAP_SORTKEY) < 240.
For example, when SET("COLLATE") == "PINYIN",
usually it can only compare 2 strings and return the correct result when the length of each string < 120.
If the length of the generated sort key (LCMAP_SORTKEY) >= 480,
there is no error, but it will only compare the left 239 bytes of the generated sort key (LCMAP_SORTKEY).
If the length of the generated sort key (LCMAP_SORTKEY) >= 240,
it will only compare the left 239 bytes of the generated sort key (LCMAP_SORTKEY).
To compare large strings in SQL statements, recommend to use the expression:
fld1=x1 AND fld2=x2 AND fld3=x3
instead of the expression:
fld1+fld2+fld3=x1+x2+x3
3. RESOLUTION:
We can write some code to fix the BUG.
Fun53beb2 :: ; proc near
push ebp ;0x0053beb2 : 55
mov ebp , esp ;0x0053beb3 : 8bec
sub esp , 020h ;0x0053beb5 : 83ec20
mov eax , dword ptr [ Data9370f8 ] ;0x0053beb8 : a1f8709300
mov edx , dword ptr [eax] ;0x0053bebd : 8b10
mov ecx , dword ptr [ ebp + 12 ] ;0x0053bebf : 8b4d0c
mov eax , dword ptr [ edx + 0C8h ] ;0x0053bec2 : 8b82c8000000
test eax , eax ;0x0053bec8 : 85c0
push ebx ;0x0053beca : 53
push esi ;0x0053becb : 56
mov esi , dword ptr [ ebp + 8 ] ;0x0053becc : 8b7508
mov ebx , dword ptr [ esi + 8 ] ;0x0053becf : 8b5e08
push edi ;0x0053bed2 : 57
mov edi , dword ptr [ ecx + 8 ] ;0x0053bed3 : 8b7908
mov dword ptr [ ebp - 16 ] , edi ;0x0053bed6 : 897df0
mov dword ptr [ ebp - 8 ] , eax ;0x0053bed9 : 8945f8
je Label502873 ;0x0053bedc : 0f849169fcff
cmp dword ptr [ esi + 4 ] , 01h ;0x0053bee2 : 837e0401
je Label502873 ;0x0053bee6 : 0f848769fcff
;
; ----------------------------------------------------
; VFP 9.0 FIX - COMPARE LARGE STRINGS
; May 2022
; ----------------------------------------------------
; CCB
;
; Compare large strings.
;
; 2022/5/10, by ccb
;
cmp dword ptr vfpa_sys9078_data,00h
je Label53beeb
cmp ebx , 07FFFE0h
jge Label58e6b4
cmp edi , 07FFFE0h
jge Label58e6b4
lea eax , dword ptr [ ebx + edi ]
cmp eax , 10000h
jle Label53beeb
mov dword ptr [ ebp - 12 ] , ebx
mov dword ptr [ ebp - 28 ] , edi
mov dword ptr [ ebp - 24 ] , 00h
mov dword ptr [ ebp - 20 ] , 00h
jmp Label58e685
Label53beeb ::
lea eax , dword ptr [ ebx + edi ] ;0x0053beec : 8d043b
add eax , 03h ;0x0053beef : 83c003
and eax , 0FFFFFFFCh ;0x0053bef2 : 83e0fc
mov dword ptr [ ebp - 12 ] , ebx ;0x0053bef5 : 895df4
mov dword ptr [ ebp - 28 ] , edi ;0x0053bef8 : 897de4
call Fun42c118 ;0x0053befb : e81802efff
mov ecx , esp ;0x0053bf00 : 8bcc
test ecx , ecx ;0x0053bf02 : 85c9
mov dword ptr [ ebp - 24 ] , ecx ;0x0053bf04 : 894de8
je Label58e685 ;0x0053bf07 : 0f8478270500
mov eax , dword ptr [ esi + 32 ] ;0x0053bf0d : 8b4620
mov edx , dword ptr [eax] ;0x0053bf10 : 8b10
push ebx ;0x0053bf12 : 53
mov dword ptr [ ebp - 20 ] , 01h ;0x0053bf13 : c745ec01000000
call Fun42c19b ;0x0053bf1a : e87c02efff
mov ecx , dword ptr [ ebp + 12 ] ;0x0053bf1f : 8b4d0c
mov eax , dword ptr [ ebp - 24 ] ;0x0053bf22 : 8b45e8
mov edx , dword ptr [ ecx + 32 ] ;0x0053bf25 : 8b5120
lea ecx , dword ptr [ eax + ebx ] ;0x0053bf28 : 8d0c18
Label53bf2b ::
mov edx , dword ptr [edx] ;0x0053bf2b : 8b12
push edi ;0x0053bf2d : 57
call Fun42c19b ;0x0053bf2e : e86802efff
mov eax , dword ptr [ ebp + 16 ] ;0x0053bf33 : 8b4510
sub eax , 02h ;0x0053bf36 : 83e802
je Label58e72b ;0x0053bf39 : 0f84ec270500
cmp ebx , edi ;0x0053bf3f : 3bdf
mov ecx , ebx ;0x0053bf41 : 8bcb
jb Label53bf47 ;0x0053bf43 : 7202
mov ecx , edi ;0x0053bf45 : 8bcf
Label53bf47 ::
mov edx , dword ptr [ ebp + 12 ] ;0x0053bf47 : 8b550c
mov eax , dword ptr [ edx + 32 ] ;0x0053bf4a : 8b4220
mov edx , dword ptr [ esi + 32 ] ;0x0053bf4d : 8b5620
mov edi , dword ptr [eax] ;0x0053bf50 : 8b38
mov esi , dword ptr [edx] ;0x0053bf52 : 8b32
xor eax , eax ;0x0053bf54 : 33c0
repz cmpsb ;0x0053bf56 : f3a6
je Label53bf5f ;0x0053bf58 : 7405
sbb eax , eax ;0x0053bf5a : 1bc0
sbb eax , 0FFFFFFFFh ;0x0053bf5c : 83d8ff
Label53bf5f ::
mov edi , eax ;0x0053bf5f : 8bf8
test edi , edi ;0x0053bf61 : 85ff
je Label53bfb9 ;0x0053bf63 : 7454
mov esi , dword ptr [ ebp - 8 ] ;0x0053bf65 : 8b75f8
mov edi , dword ptr [ ebp + 8 ] ;0x0053bf68 : 8b7d08
push esi ;0x0053bf6b : 56
xor eax , eax ;0x0053bf6c : 33c0
mov ecx , edi ;0x0053bf6e : 8bcf
call Fun53c1a9 ;0x0053bf70 : e834020000
push esi ;0x0053bf75 : 56
mov esi , dword ptr [ ebp + 12 ] ;0x0053bf76 : 8b750c
mov ebx , eax ;0x0053bf79 : 8bd8
xor eax , eax ;0x0053bf7b : 33c0
mov ecx , esi ;0x0053bf7d : 8bce
call Fun53c1a9 ;0x0053bf7f : e825020000
mov ecx , eax ;0x0053bf84 : 8bc8
mov eax , dword ptr [ ebp - 8 ] ;0x0053bf86 : 8b45f8
mov edx , dword ptr [eax] ;0x0053bf89 : 8b10
mov eax , dword ptr [ edx + 16 ] ;0x0053bf8b : 8b4210
test eax , eax ;0x0053bf8e : 85c0
mov dword ptr [ ebp - 16 ] , ecx ;0x0053bf90 : 894df0
jne Label58e716 ;0x0053bf93 : 0f857d270500
cmp ebx , ecx ;0x0053bf99 : 3bd9
pushd 00h ;0x0053bf9b : 6a00
jb Label53b85e ;0x0053bf9d : 0f82bbf8ffff
Label53bfa3 ::
mov edx , dword ptr [ esi + 32 ] ;0x0053bfa3 : 8b5620
mov eax , dword ptr [edx] ;0x0053bfa6 : 8b02
push eax ;0x0053bfa8 : 50
mov eax , dword ptr [ edi + 32 ] ;0x0053bfa9 : 8b4720
mov eax , dword ptr [eax] ;0x0053bfac : 8b00
mov edx , ecx ;0x0053bfae : 8bd1
mov ecx , ebx ;0x0053bfb0 : 8bcb
call Fun53c1fb ;0x0053bfb2 : e844020000
mov edi , eax ;0x0053bfb7 : 8bf8
Label53bfb9 ::
mov esi , dword ptr [ ebp + 8 ] ;0x0053bfb9 : 8b7508
Label53bfbc ::
mov eax , dword ptr [ ebp - 20 ] ;0x0053bfbc : 8b45ec
test eax , eax ;0x0053bfbf : 85c0
mov eax , dword ptr [ esi + 32 ] ;0x0053bfc1 : 8b4620
je Label58e768 ;0x0053bfc4 : 0f849e270500
mov ecx , dword ptr [ ebp - 12 ] ;0x0053bfca : 8b4df4
mov esi , dword ptr [ ebp - 24 ] ;0x0053bfcd : 8b75e8
push ecx ;0x0053bfd0 : 51
mov ecx , dword ptr [eax] ;0x0053bfd1 : 8b08
mov edx , esi ;0x0053bfd3 : 8bd6
call Fun42c19b ;0x0053bfd5 : e8c101efff
mov ecx , dword ptr [ ebp - 28 ] ;0x0053bfda : 8b4de4
mov eax , dword ptr [ ebp + 12 ] ;0x0053bfdd : 8b450c
mov edx , dword ptr [ ebp - 12 ] ;0x0053bfe0 : 8b55f4
push ecx ;0x0053bfe3 : 51
mov ecx , dword ptr [ eax + 32 ] ;0x0053bfe4 : 8b4820
mov ecx , dword ptr [ecx] ;0x0053bfe7 : 8b09
add edx , esi ;0x0053bfe9 : 03d6
call Fun42c19b ;0x0053bfeb : e8ab01efff
Label53bff0 ::
test edi , edi ;0x0053bff0 : 85ff
je Label53b7d7 ;0x0053bff2 : 0f84dff7ffff
Label53bff8 ::
mov eax , edi ;0x0053bff8 : 8bc7
lea esp , dword ptr [ ebp - 44 ] ;0x0053bffa : 8d65d4
pop edi ;0x0053bffd : 5f
pop esi ;0x0053bffe : 5e
pop ebx ;0x0053bfff : 5b
mov esp , ebp ;0x0053c000 : 8be5
pop ebp ;0x0053c002 : 5d
ret 0Ch ;0x0053c003 : c20c00
4. APPLIES TO:
VFP 6.0.8167.0
VFP 6.0.8961.0 (SP5)
VFP 7.0.0.9262
VFP 7.0.0.9465 (SP1)
VFP 8.0.0.2521
VFP 8.0.0.3117 (SP1)
VFP 9.0.0.2412
VFP 9.0.0.3504 (SP1)
VFP 9.0.0.4611 (SP2)
VFP 9.0.0.5015 (SP2)
VFP 9.0.0.5411 (SP2)
VFP 9.0.0.5721 (SP2)
VFP 9.0.0.5815 (SP2)
VFP 9.0.0.6303 (SP2)
VFP 9.0.0.6602 (SP2)
VFP 9.0.0.7423 (SP2)
The bug has been fixed in VFP Advanced.
5. REFERENCE WEBSITES:
1, baiyujia.com:
http://www.baiyujia.com
http://www.baiyujia.com/vfpdocuments/f_vfp9fix210.asp
http://www.baiyujia.com/vfpdocuments/f_vfp9fix208.asp
http://www.baiyujia.com/vfpdocuments/f_vfp9fix209.asp
6. OTHER:
For reference only, there is no guarantees.
Any questions or suggestions, please send me an email at ccb2000@163.com.
|