[apache-hb]

Resolves #152. I opted to use the compiler provided extension support rather than modify codec_choose_x86 just to be sure I didn't introduce any bugs into the autodetection logic. I've tested these changes on zen4 Genoa windows & linux, as well as qemu without kvm enabled. Changes were tested on clang, gcc, and msvc.

[aklomp]

This might not be very portable and it seems to be breaking CI. Perhaps it can be wrapped in an #ifdef __has_builtin?

[aklomp]

While I agree that it's bad form for the tests to crash without warning, I'm not totally sure about this solution yet. Two reasons:

• There is already some "mature" runtime feature detection code in lib/codec_choose.c, and this PR feels like it's treading the same ground. I'd prefer to unify those things if possible. (I'm not happy with the runtime detection code there by the way - it's a wart that grew out of necessity and not a problem I want to have.)
• Catching SIGILL seems like possibly a reasonable way to deal with the issue, since these are just tests.
  • Catching signals portably across platforms is another can of worms.

[apache-hb]

Sorry I couldn't respond sooner, I've been very busy at my day job. Your comments make sense, I was initialy hesitant to use the internals of codec_choose_x86 because I didn't want to accidentally export a symbol from it that wasnt previously visible. If I can figure out how to ensure no symbols leak I'll push a new revision using that function to detect feature support rather than using compiler builtins.

[mcourteaux]

@aklomp Is this PR ready to land? Can you approve CI?

[aklomp]

@aklomp Is this PR ready to land? Can you approve CI?

CI is approved, sorry for the delay. Unfortunately it's failing.

I did a quick first review on the revised code and I like the approach. It feels clean to separate the feature detection code from the codec choosing code.

There are some things that I'm not fond of, like removing some of the #if HAVE_xxx guards. Those are functional. They prevent code from being compiled for platforms where the code is not needed, and they also prevent auto-selecting a codec that is not present in the build.

What I think can happen now is the following. Let's say that the library is compiled at SSE4 support level, but run on an AVX2 machine. The new codec chooser picks the AVX2 codec, but that one won't work because it's a no-op in that build. The right behavior would be to fall back to SSE4. The if HAVE_xxx logic was there to limit the handled support level to just the codecs that are built in.

Ironically, I think this is similar to the #152 issue that started the discussion?

[mcourteaux]

Thanks for taking a look! I think there are two options indeed:

1. Always compile in all the versions, and let the runtime detect which version is appropriate. Potentially a less lean binary, but simple.
2. Let the user (or CMake) decide which versions to compile, and have the runtime not select what's not available. This is what you're saying, and offers maximal control to the user

I'd also be in favor of option 2. However, I'd like to argue that HAVE_AVX2 is too generic of a name, and can cause bugs in larger projects who use this project as a subproject in CMake. A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily. To that end, I propose getting rid of those aliases defined here:
https://github.com/aklomp/base64/blob/9e8ed65048ff0f703fad3deb03bf66ac7f78a4d7/cmake/config.h.in#L4C1-L27C1
and adjusting the necessary code in the project.

I could do that and make my own PR to replace this one? Or you could do it? Any preferences?

[BurningEnlightenment]

A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily.

Can you describe a realistic scenario for this? I can only think of global unity builds (where you very likely also get in trouble due to non-namespaced static globals like codec) or someone doing bad practice stuff like abusing CMAKE_C_FLAGS by explicitly adding -DHAVE_AVX=1.

[mcourteaux]

A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily.

Can you describe a realistic scenario for this? I can only think of global unity builds (where you very likely also get in trouble due to non-namespaced static globals like codec) or someone doing bad practice stuff like abusing CMAKE_C_FLAGS by explicitly adding -DHAVE_AVX=1.

I think a simple and commonly overused add_definition(HAVE_AVX) will already do the trick.

[BurningEnlightenment]

I think a simple and commonly overused add_definition(HAVE_AVX) will already do the trick.

Which does only matter if this project is part of the directory scope calling add_definition. ExternalProject_Add opens a new independent directory scope. This leaves people who call add_subdirectory on external projects (i.e. this project) which is just asking for trouble in any case.

[apache-hb]

I think this should resolve the linking issues, I moved the feature detection to its own translation unit so the tests can build it seperately and get access to the feature level since its not in the public exports of the base64 library

[mvanslobbe]

Thanks for picking this up. I tested fd3602f against master on an x86 machine without AVX-512, plus Intel SDE, and the core idea works: the AVX-512 tests are now skipped natively and still run under sde -icl. Keeping the HAVE_xxx guards inside x86_get_cpu_feature_level() also preserves the fallback @aklomp was concerned about.

I did hit three regressions compared with master, though.

1. Null codec when no built-in SIMD level matches. codec_choose_x86() now always returns true. When x86_get_cpu_feature_level() returns X86_FEATURE_LEVEL_NONE, codec_choose_forced(codec, 0) sets nothing and the codec stays NULL, so the first encode calls address 0:

$ cmake -DBASE64_WITH_AVX2=OFF -DBASE64_WITH_AVX=OFF -DBASE64_WITH_SSE42=OFF \
        -DBASE64_WITH_SSE41=OFF -DBASE64_WITH_SSSE3=OFF ...   # AVX-512 only
$ echo hello | bin/base64        # on a CPU without AVX-512
Segmentation fault
#0  0x0000000000000000 in ?? ()
#1  base64_stream_encode ()

The same happens with a default build under sde -p4p (no SSSE3). Master falls back to the plain codec in both cases. This affects library users, not only the tests.

2. The plain codec is no longer tested. cpu_has_x86_extension() returns 0 for any name it does not recognise, so plain, as well as NEON32 and NEON64 on ARM, are skipped everywhere:

Codec plain:
  skipping

3. Shared builds fail to link the tests. With -DBUILD_SHARED_LIBS=ON:

codec_supported.c:(.text+0x10d): undefined reference to `x86_get_cpu_feature_level'

The symbol is not exported from the library, and only test/Makefile compiles feature_level.c into the tests. This may also be what is breaking CI.

The following fixes all three for me:

--- a/lib/codec_choose.c
+++ b/lib/codec_choose.c
@@ -136,8 +136,7 @@ codec_choose_x86 (struct codec *codec)
 	feature_level = x86_get_cpu_feature_level();
-	codec_choose_forced(codec, cpu_feature_flags[feature_level]);
-	return true;
+	return codec_choose_forced(codec, cpu_feature_flags[feature_level]);

--- a/test/codec_supported.c
+++ b/test/codec_supported.c
@@ -31,10 +31,10 @@ cpu_has_x86_extension (const char *codec)
 	if (strcmp(codec, "AVX512") == 0) {
 		return feature_level >= X86_FEATURE_LEVEL_AVX512;
 	}
-	return 0;
+	return 1;
 #else
 	(void)codec;
-	return 0;
+	return 1;
 #endif

--- a/test/CMakeLists.txt
+++ b/test/CMakeLists.txt
@@ -29,14 +29,20 @@ endfunction()
 add_base64_test(test_base64
     codec_supported.c
+    ../lib/feature_level.c
     test_base64.c
 )

 add_base64_test(benchmark
     codec_supported.c
+    ../lib/feature_level.c
     benchmark.c
 )

+foreach(_TARGET test_base64 benchmark)
+    target_include_directories(${_TARGET} PRIVATE "${PROJECT_BINARY_DIR}")
+endforeach()

With these changes, I checked:

• static and shared CMake builds, natively: all codecs the CPU supports pass, plain included, and AVX-512 is skipped;
• sde -icl: the AVX-512 tests run and pass;
• sde -p4p: only plain runs, and the CLI works;
• an AVX-512-only build, natively: the CLI falls back to plain and works;
• the Makefile build: passes.